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Foreword 


Our  academic  contributions  to  developing  future  officers  are  significant.  The  experiences  we  offer 
cadets  are  driven  by  our  institutional  outcomes,  which  give  an  important  sense  of  purpose  to  the  work 
we  do.  The  vibrant  learning-focused  culture  we  strive  to  attain  requires  feedback  mechanisms  to  ensure 
that  our  efforts  as  educators  yield  the  important  outcomes  we  set  for  our  students,  our  programs,  and 
our  institution,  A  strong  assessment  program  is  essential  to  make  this  happen. 

The  importance  of  assessment  is  not  unique  to  USAFA;  indeed,  it  is  apparent  across  the  entire  higher 
education  landscape.  Nowhere  is  this  seen  more  dearly  than  in  the  latest  accreditation  criteria 
published  by  the  Higher  Learning  Commission  (HLt),  which  dearly  state  that  institutions  must  assess 
"achievement  of  the  learning  outcomes  that  it  claims  for  its  curricular  and  co-curricular  programs"  and 
then  use  "the  information  from  assessment  to  improve  student  learning"  (Core  Component  4,B.). 

Prior  to  becoming  the  Dean  of  the  Faculty,  I  had  the  opportunity  to  visit  each  academic  department  and 
learn  about  the  assessment  processes  they  currently  have  in  place.  It  quickly  became  apparent  that 
there  is  substantial  variability  in  the  maturity  of  different  assessment  programs.  Some  programs  are  very 
mature,  demonstrating  well-established  processes  for  collecting  and  making  sense  of  assessment  data 
and  using  them  to  improve  the  experiences  they  offer  cadets,  both  in  and  out  of  the  classroom.  Others 
are  less  far  along,  still  working  to  clarify  the  desired  learning  outcomes  of  their  academic  major  and 
mapping  those  outcomes  to  their  curriculum.  That  variability  is  understandable.  However,  my  intent  is 
to  ensure  that  all  of  our  programs'  assessment  efforts  reach  the  more  mature  level.  Doing  so  will  allow 
us  to  meet  the  expectations  of  the  HLC  and  will  also  promote  a  more  effective  learning-focused  culture. 

That  is  why  1  see  this  Assessment  Handbook  as  a  valuable  contribution  to  the  DF  community,  ft  serves  as 
a  guide  that  faculty  across  DF  can  use  in  developing,  implementing,  and  improving  assessment  processes 
in  all  academic  majors  and  minors.  Regardless  of  where  your  own  assessment  program  currently  stands, 

I  encourage  you  to  use  this  guide  to  help  make  it  better.  By  doing  so,  we  can  alt  ensure  that  the 
educational  experiences  offered  to  our  cadets  are  as  effective  as  they  can  possibly  be. 


Brigadier  General  Andrew  P.  Armacost 
Dean  of  the  Faculty 
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Chapter  1  -  USAFA's  Focus  on  Cadet  Learning1 

In  the  past  two  decades,  both  the  Air  Force  Academy  and  the  higher  education  community  at  large  have 
experienced  a  significant  paradigm  shift,  from  a  traditional  "instruction  focus"  to  a  newer  "learning 
focus"  (Barr  &  Tagg,  1995;  Huba  &  Freed,  2000;  Tagg,  2003).  According  to  the  older  instruction-focused 
paradigm  (in  which  many  of  us  grew  up),  faculty  members'  primary  teaching-related  task  was  to  deliver 
instruction -to  transmit  their  disciplinary  knowledge  to  students.  In  this  view,  successful  faculty 
members  were  those  who  had  the  most  up-to-date  knowledge  of  their  discipline  and  who  were  able  to 
transmit  that  knowledge  in  ways  that  were  both  clear  and  well-organized.  There  was  obviously  a  hope 
that  students  benefited  from  the  process,  but  identifying  the  exact  nature  of  that  benefit  was  generally 
not  emphasized. 

More  recently,  faculty  members  have  come  to  realize  a  fundamental  problem  with  the  instruction- 
focused  paradigm.  It  is  possible  for  faculty  members  to  deliver  instruction  -  and  perhaps  even  deliver  it 
really  well  -  without  it  having  much  of  an  impact  on  students*  You  may  have  experienced  this 
phenomenon  yourself*  Have  you  ever  delivered  a  clear,  well-organized  lesson  within  your  discipline, 
only  to  be  disappointed  when  you  discovered  that  your  cadets  didn't  really  understand  what  you  were 
trying  to  convey?  If  so,  then  you  are  acutely  awa  re  of  the  frustrations  that  arise  when  what  you  teach 
may  not  be  the  same  as  what  your  cadets  learn. 

This  brings  us  to  the  newer  learning  focus  -  and  the  emphasis  on  assessment  that  comes  with  it.  Rather 
than  being  content  with  the  mere  delivery  of  various  units  of  instruction,  colleges  and  universities  {to 
include  USAFA)  are  being  called  upon  to  consider  the  impact  of  that  instruction.  What  are  students 
really  learning  in  our  courses  and  programs?  Are  they  accomplishing  what  we  want  them  to  accomplish? 
And,  in  those  situations  where  they  are  not,  what  can  we  do  to  improve  so  that  their  learning  improves? 

Where  Did  The  Learning  Focus  Come  From? 

Many  factors  have  contributed  to  the  heightened  emphasis  on  student  learning,  both  here  at  USAFA  and 
nationwide.  One  obvious  factor  is  the  rising  cost  of  a  college  degree.  According  to  the  College  Board 
(http://trend5xolleKeboard.Qrg/college  pricing),  tuition  costs  at  public  four-year  institutions  have 
grown  by  an  average  of  over  5%  annually  over  the  past  ten  years,  a  figure  which  has  consistently 

1  Some  of  the  information  in  this  chapter  comes  from  Jones,  Noyd,  &  Sagendorf  s  (2014)  Building  a  teaming 
Pathway  to  Student  Learning:  A  How-to  Guide  to  Course  Design  (Stylus  Publishing),  which  describes  the  origin  of 
the  learning  focus  in  significantly  more  depth. 
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exceeded  the  rate  of  inflation.  This  has  led  writers  and  policymakers  to  question  whether  the  education 
students  receive  is  really  worth  the  growing  price  tag;  the  answers  haven't  always  been  pretty.  For 
instance,  under  the  George  W.  Bush  administration,  the  Secretary  of  Education's  Commission  on  the 
Future  of  Higher  Education  (i.e.,  the  "Spellings  Commission")  released  a  scathing  review  of  higher 
education  practices  (U.S*  Department  of  Education,  2006)  noting  a  "remarkable  absence  of 
accountability  mechanisms  to  ensure  that  colleges  succeed  in  educating  students."  (p.  vii).  Since  that 
time,  Derek  Bok's  (2006)  Our  Underachieving  Colleges ,  Arum  &  Roksa's  (2011)  Academically  Adrift,  and 
Keeling  &  Hersh's  (2011)  We're  Losing  Our  Minds:  Rethinking  American  Higher  Education  have  achieved 
popular  success  by  pointing  out  areas  in  which  colleges  and  universities  appear  to  be  falling  short.  There 
appears  to  be  increasing  public  concern  about  the  quality  of  student  learning  in  college,  and  those 
concerns  aren't  likely  to  go  away  unless  (and  until)  college  faculty  and  staff  respond  by  adopting  a  more 
intentional  approach  to  promoting  and  assessing  student  learning. 

A  related  concern  has  been  raised  in  national  conversations  about  the  purpose  of  a  college  education  in 
the  first  place  (e.g,,  Association  of  American  Colleges  &  Universities,  2007;  Summers,  2012),  What  do  we 
really  want  students  to  learn  from  their  college  experience?  What  do  we  want  to  be  true  of  those 
people  who  attend  and  graduate  from  our  colleges  and  universities  that  is  not  true  of  those  who  don't? 
Is  mastering  disparate  pieces  of  disciplinary  content  (like  what  has  traditionally  been  delivered  in  our 
classes)  really  what  we  are  looking  for?  Or  is  college  supposed  to  be  about  something  more? 

The  answers  that  are  emerging  in  this  national  conversation  are  intriguing.  Despite  the  fact  that  most 
college  courses  have  historically  been  organized  around  the  content  of  a  particular  discipline,  many  of 
the  things  people  ultimately  want  students  to  learn  in  college  do  not  necessarily  seem  to  be  tied  to 
specific  disciplinary  knowledge.  For  example,  in  their  published  set  of  "essential  learning  outcomes"  for 
21s'  century  students,  the  Association  of  American  Colleges  and  Universities  (AAC&U,  2007)  stresses  the 
importance  of  general-purpose  "intellectual  and  practical  skills,"  such  as  critical  thinking  and  written 
communication;  "personal  and  social  responsibilities/1  such  as  intercultural  competence  and  ethical 
reasoning;  and  "integrative  and  applied  learning"  that  allows  students  to  build  strong  connections 
across  disciplinary  boundaries.  Accomplishing  these  lofty  goals  isn't  likely  to  happen  by  accident  -  and  it 
almost  certainly  won't  happen  if  our  programs  are  based  solely  on  delivery  of  disciplinary  knowledge. 
Instead,  they  will  require  intentional  effort  on  the  part  of  faculty  and  staff,  as  well  as  a  heightened 
awareness  about  the  nature  and  quality  of  our  students'  learning. 
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The  Learning  Focus  at  U5AFA 

Here  at  the  Air  Force  Academy,  we  have  especially  important  reasons  to  focus  on  cadet  learning.  First, 
the  Air  Force  Academy  has  a  vitally  important  mission  -  to  develop  officers  of  character  motivated  to 
lead  the  Air  Force  in  service  to  the  nation  -  which  we  can  obviously  not  afford  to  take  lightly.  It  is  for 
that  reason  that  we  have  established  our  own  set  of  institutional  learning  outcomes  that  are  just  as 
rigorous  as  those  published  by  AAC&U,  and  it  will  take  a  very  intentional  effort  from  all  faculty  and  staff 
to  help  cadets  achieve  them.  Second,  as  a  federally-funded  institution,  we  are  stewards  of  valuable 
taxpayer  dollars.  As  a  result,  we  owe  it  to  the  American  public  to  use  their  money  responsibly,  making 
sure  that  we  foster  cadet  learning  to  the  very  highest  extent  possible.  Finally,  and  arguably  most 
importantly,  focusing  on  student  learning  is  simply  the  right  thing  to  do.  Here  at  USAFA,  we  have  a 
tremendous  opportunity  to  positively  impact  the  lives  of  thousands  of  America's  most  talented  young 
people  ...  why  wouldn't  we  want  to  make  sure  that  we  are  doing  that  in  the  very  best  way  we  possibly 
can? 

It  is  within  this  context  that  USAFA  has  adopted  a  "learning  focus"  that  calls  us  to  (1)  be  very  clear  about 
what  we  want  cadets  to  learn;  (2)  design  learning  experiences  (both  in  and  outside  the  classroom)  that 
are  well-aligned  with  those  learning  goals;  (3)  collect  assessment  data  that  inform  us  about  the  extent  to 
which  cadets  are  meeting  the  goals;  (4)  make  sense  of  the  assessment  data;  and  (5)  use  what  we've 
learned  to  continually  improve  our  programs  in  areas  where  we  may  be  falling  short.  This  can  be 
depicted  visually  in  Figure  1,  which  is  a  slight  variant  of  an  image  disseminated  throughout  the  Academy 
over  the  past  several  years. 
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Figure  1.  The  learning  focus 


In  addition  to  framing  the  assessment  efforts  at  the  institutional  level,  this  learning-foeused  model  is 
also  an  effective  means  of  framing  the  more  specific  assessment  efforts  within  your  academic  major  or 
minor.  That  is,  an  effective  assessment  program  in  your  program  will  have  each  of  the  elements  of  the 
model,  to  include: 

(1)  Outcomes  for  cadet  learning  in  your  academic  major  or  minor; 

(2)  A  set  of  learning  experiences  (potentially  to  include  both  academic  courses  and  extracurricular 
programs)  that  help  cadets  achieve  those  cadet  learning  outcomes; 

(3)  A  strategy  for  collecting  assessment  data  showing  the  extent  to  which  cadets  have  achieved 
your  learning  outcomes; 

(4)  A  means  of  making  sense  of  the  assessment  data  that  have  been  collected;  and 

(5)  A  process  that  allows  your  department  or  program  to  make  improvements  based  on  what  you 


learn. 


This  Assessment  Handbook  was  written  to  provide  guidance  to  departmental  assessment 
representatives  and  other  faculty  members  who  are  charged  with  leading  assessment  efforts  in  their 
respective  academic  programs.  It  is  a  "how-to"  book,  meant  to  guide  you  through  the  steps  of  the 
learning-focused  model  as  they  pertain  to  your  academic  assessment  program.  Along  the  way,  I  will  also 


10 


highlight  some  of  the  best  assessment  practices  that  are  currently  in  place  across  the  faculty.  The  hope 
is  that  this  document  can  help  all  of  us  learn  from  each  other,  so  that  we  can  all  engage  more  effectively 
in  the  dynamic  process  of  assessment  and  improvement. 

Organization  of  the  Handbook 

This  Assessment  Handbook  is  divided  into  two  distinct  parts.  In  Part  I  (Le.,  Chapters  1  &  2),  the  emphasis 
is  on  why  it  is  important  for  each  of  our  academic  majors  and  minors  to  engage  in  assessment  of  cadet 
learning.  As  you've  seen,  the  most  important  reasons  to  engage  in  assessment  have  to  do  with  our 
shared  commitment  to  improving  cadet  learning.  However,  it  is  important  to  recognize  that 
accreditation  plays  a  motivating  factor  as  well.  Therefore,  in  Chapter  2,  I  introduce  the  accreditation 
process  and  show  why  the  requirements  of  USAFA's  institutional  accrediting  body  (i.e.,  the  Higher 
Learning  Commission  of  the  North  Central  Association)  make  it  imperative  that  you  and  your  faculty 
colleagues  seriously  engage  in  the  process  of  assessment  and  improvement. 

In  Part  II  {Chapters  3-7),  the  emphasis  shifts  from  the  why  of  assessment  to  how  assessment  can  be 
done.  The  learning-focused  model  shown  in  Figure  1  serves  as  the  organizational  framework  for  this  part 
of  the  Handbook ,  Specifically; 

*  Chapter  3  is  dedicated  to  helping  you  create  effective  learning  outcomes  for  the  cadets  in  your 
academic  major  or  minor.  The  chapter  begins  by  describing  the  characteristics  of  well-written 
cadet  learning  outcomes;  the  second  half  of  the  chapter  includes  guidance  regarding  how  to 
work  with  faculty  in  your  department  to  generate  effective  cadet  learning  outcomes  for  your 
academic  major  or  minor. 

*  Chapter  4  is  designed  to  help  you  build  a  "curriculum  map"  showing  the  relationship  between 
your  cadet  learning  outcomes  and  the  various  courses  in  your  academic  major  or  minor.  As  in 
Chapter  3,  this  chapter  will  include  some  tips  on  making  effective  curriculum  maps,  and  it  will 
also  include  some  examples  of  curriculum  maps  currently  in  use  at  USAFA. 

*  Chapter  5  is  focused  on  methods  of  collecting  assessment  data.  This  chapter  is  perhaps  best 
thought  of  as  a  catalog  of  different  assessment  methodologies,  each  with  its  own  distinctive  set 
of  strengths  and  weaknesses.  See  this  chapter  for  techniques  -  many  of  which  are  already  in 
place  somewhere  here  at  the  Academy -for  determining  the  extent  to  which  cadets  are  actually 
achieving  your  program's  cadet  learning  outcomes. 


11 


•  Chapter  6  is  all  about  making  sense  of  your  assessment  results.  This  chapter  will  highlight  some 
of  the  processes  that  departments  currently  have  in  place,  and  it  also  includes  two  reporting 
templates  that  may  be  helpful  if  faculty  in  your  department  are  looking  for  a  place  to  start, 

•  Finally,  Chapter  7  is  designed  to  help  you  use  the  information  from  your  assessments  to  make 
improvements  in  your  program.  This  is  the  last  chapter  in  the  handbook,  and  it  includes  far 
fewer  technical  details  than  some  of  the  earlier  chapters  do.  However,  it  is  a  rguably  the  most 
important  chapter  in  this  entire  document.  That  is  because  doing  assessment  is  valuable  only  to 
the  extent  that  the  data  are  used  to  make  your  courses  and  programs  better.  As  such,  this 
chapter  pays  homage  to  what  George  Kuh  and  his  colleagues  (2005)  refer  to  as  "positive 
restlessness,"  an  ethos  of  constant  improvement  that  permeates  high-performing  educational 
institutions. 
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Chapter  2  -  Assessment  and  Accreditation 

In  general,  accreditation  refers  to  a  means  by  which  external  agencies  assure  the  quality  of  education 
being  provided  at  a  given  institution.  In  the  United  States,  that  accreditation  can  take  on  two  forms. 
Higher  education  institutions  are  accredited  by  one  of  six  regional  accrediting  organizations  recognized 
by  the  U.S.  Department  of  Education  and  the  Council  for  Higher  Education  Accreditation,  with  each 
organization  being  responsible  for  overseeing  degree-granting  institutions  in  a  given  portion  of  the 
country  (http://www.chea.org/Directories/reEional.asD).  In  some  cases,  academic  programs  can  also  be 
accredited  by  discipline-specific  accrediting  bodies.  Perhaps  the  most  familiar  example  of  this  kind  of 
specialized  accreditation  is  the  American  Bar  Association,  which  accredits  law  schools  and  administers 
the  bar  exam  to  prospective  lawyers. 

Here  at  the  Academy,  AFI  36-3501  mandates  that  we  maintain  both  our  institutional  accreditation  and 
other  appropriate  division  or  discipline-specific  accreditation  when  they  are  available.  Our  institutional 
accreditation  is  maintained  through  the  Higher  Learning  Commission  of  the  North  Central  Association  of 
Colleges  and  Schools  (nttp://ncahlc.orfi/l.  The  following  academic  majors  at  USAFA  also  have  the  more 
specific  programmatic  accreditation: 

•  All  of  the  Engineering  majors,  as  well  as  Computer  Science,  are  accredited  by  ABET  (previously 
named  the  Accreditation  Board  for  Engineering  and  Technology;  http://www.abei.orfi/). 

•  The  Management  major  is  accredited  by  The  Association  to  Advance  Collegiate  Schools  of 
Business  (AACSB  International;  http://www.aacsb.edu/). 

•  The  Chemistry  major  is  approved  by  the  American  Chemical  Society  (ACS; 
http://portal.acs.orfi/portal/aes/corfi/content) . 

All  of  USAFA's  accrediting  bodies  have  published  expectations  they  have  of  member  institutions  with 
regard  to  assessment  and  improvement.  For  the  Higher  Learning  Commission  (2012),  those  expectations 
are  outlined  most  clearly  in  Criterion  Four,  entitled  "Teaching  and  Learning:  Evaluation  and 
Improvement,"  specifically  within  Core  Component  4.B.,  shown  below: 


13 


Core  Component  4, B.  The  institution  demonstrates  a  commitment  to  educational  achievement  and 
improvement  through  ongoing  assessment  of  student  learning. 

1.  The  institution  has  clearly  stated  goals  for  student  learning  and  effective  processes  for 
assessment  of  student  learning  and  achievement  of  learning  goals. 

2.  The  institution  assesses  achievement  of  the  learning  outcomes  that  it  claims  for  its  curricular  and 
co-curricular  programs . 

3 .  The  institution  uses  the  information  from  assessment  to  improve  student  learning. 

4 .  The  institution's  processes  and  methodologies  to  assess  student  learning  reflect  good  practice , 
including  the  substantial  participation  of  faculty  and  other  instructional  staff  members. 

In  order  to  maintain  its  status  as  an  accredited  institution,  the  Academy  is  required  to  submit  both  an 
Assurance  Argument  and  an  Evidence  File  to  the  Higher  Learning  Commission2.  The  Assurance  Argument 
is  a  relatively  short  narrative  (limited  to  no  more  than  35,000  words)  that  presents  the  Academy's  case 
for  how  we  meet  each  of  the  core  components.  The  Evidence  File  consists  of  those  documents  that  are 
referred  to  in  the  Assurance  Argument. 

What  does  this  mean  for  you?  Take  a  close  look  at  the  items  fisted  in  Core  Component  4. B.  This  core 
component  essentially  lays  out  the  same  expectations  that  we  described  in  Chapter  1.  That  is,  faculty 
from  each  curricular  program  (Lev  academic  major  or  minor)  are  expected  to: 

(1)  Articulate  the  cadet  learning  outcomes  for  the  major  or  minor; 

(2}  Identify  the  learning  experiences  (potentially  to  include  both  academic  courses  and 
extracurricular  programs)  that  help  cadets  achieve  those  learning  outcomes; 

(3)  Develop  a  strategy  for  assessing  the  extent  to  which  cadets  have  achieved  those  learning 
outcomes; 

(4)  Create  a  method  of  making  sense  of  the  assessment  results;  and 

(5)  Use  a  process  that  allows  them  to  make  improvements  based  on  what  is  learned. 

In  other  words,  in  order  to  maintain  institutional  accreditation,  faculty  from  each  academic  major  and 
minor  will  need  to  successfully  enact  the  various  steps  of  the  teaming-focused  cycle. 


2  USAFA  is  on  a  ten-year  accreditation  cycle;  the  Higher  Learning  Commission's  next  review  of  our  Assurance 
Argument  and  Evidence  File  will  take  place  in  the  Spring  of  2019. 
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An  Important  Caveat 

While  assessment  is  vitally  important  to  our  continued  accreditation,  it  is  important  to  remember  that 
assessment  isn't  something  we  do  solely  because  of  accreditation.  At  the  Air  Force  Academy,  the  most 
important  reason  we  do  assessment  is  because  of  our  shared  commitment  to  cadet  learning.  We  all 
recognize  the  vital  mission  we  have  to  educate,  train,  and  inspire  cadets  to  become  leaders  of  character, 
and  assessment  is  an  important  way  that  we  can  improve  our  ability  to  accomplish  that  mission.  In  that 
regard,  assessment  has  much  more  to  do  with  our  commitment  than  it  does  with  our  need  for 
compliance  with  external  demands. 

To  further  reinforce  the  idea  that  the  assessment  efforts  play  an  important  role  in  your  department's 
culture,  we  have  created  several  "Principles  of  Assessment-Based  Improvement"  to  help  guide  your 
approach3.  Those  principles  are  listed  here: 

Within  USAFA  departments  and  academic  programs,  assessment ... 

1.  Guides  continuous  improvement  of  our  ability  to  foster  achievement  of  the  learning  outcomes. 

•  Assessment  is  not  an  end  in  itself,  but  a  vehicle  for  improving  cadets'  learning  experiences. 
Assessment  is  useful  to  the  extent  that  it  helps  decision  makers  in  their  efforts  to  make  the 
department  or  program  better. 

•  Assessment  cultivates  a  "culture  of  evidence"  that  guides  decisions  throughout  the  department 
or  program. 

2.  Recognizes  the  intentionally  developmental  nature  of  the  cadet  experience. 

•  Learning  is  a  complex  activity  that  takes  place  over  time.  Therefore,  we  fully  expect  that  cadets 
will  demonstrate  growth  and  development  over  the  entire  span  of  their  academic  program. 

•  To  facilitate  this  development,  our  curricula  should  be  intentionally  designed  to  guide  cadets 
through  increasingly  complex  levels  of  learning. 

•  Recognizing  that  the  level  of  cadet  accomplishment  at  one  point  in  a  cadet's  career  will  be 
different  than  the  level  of  accomplishment  at  a  second  point  in  a  cadet's  career,  assessment  of 


3  These  principles  are  borrowed  heavily  from  the  AAHE  Assessment  Forum  (1996},  the  Higher  Learning 
Commission  (2003),  and  the  Association  of  American  Colleges  &  Universities  (2008}. 
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cadet  learning  will  ideally  occur  multiple  times  over  the  course  of  cadets'  time  in  our  majors  and 
minors. 

3.  Is  a  shared  responsibility. 

•  Promoting  cadet  development  is  the  Academy's  raison  d'etre.  Everyone  has  a  stake  in  it,  and 
everyone  has  a  stake  in  seeing  it  improve. 

•  As  a  result,  assessment  is  not  a  task  (or  set  of  tasks)  done  by  the  Director  of  Academic 
Assessment,  nor  is  it  a  task  solely  for  your  Departmental  Assessment  representative.  It  is  a 
collaborative  activity  engaged  in  by  all  faculty,  staff,  and  cadets, 

•  Assessment  also  involves  important  constituencies  beyond  the  Academy,  to  include  alumni, 
supervisors  in  the  field,  and  the  larger  Air  Force.  These  groups  can  provide  valuable  feedback 
about  the  quality  of  our  graduates,  as  well  as  a  stronger  sense  of  the  appropriateness  of  our 
cadet  learning  outcomes.  In  Chapter  5,  you  will  read  more  about  how  some  USAFA  programs  are 
leveraging  their  connections  with  the  graduate  community  to  gather  valuable  assessment 
information. 

4,  Is  itself  subject  to  critical  examination  and  continual  improvement*. 

•  Programmatic  assessment  plans  are  intended  to  be  living  documents,  and  assessment 
procedures  should  be  regularly  reviewed  and  updated.  No  assessment  program  is  perfect,  and 
we  can  all  learn  from  each  other  how  to  make  our  programs  better. 

•  Assessment  structures  and  processes  are  routinely  benchmarked  against  peer  institutions  and 
the  state-of-the-art  in  higher  education. 


a  As  evidence  of  the  dynamic  nature  of  assessment  structures  and  processes,  it  is  important  to  note  that  the 
accreditation  criteria  used  by  the  Higher  Learning  Commission  changed  in  2012.  Therefore,  the  criteria  against 
which  USAFA  will  be  judged  during  our  next  comprehensive  evaluation  will  be  different  than  the  ones  used  during 
our  last  accreditation  visit. 
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Part  II:  HOWto  Create  (or  Refine)  an  Assessment 
Program  in  Each  Academic  Major  and  Minor 
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Chapter  3  -  Creating  Cadet  Learning  Outcomes 

Imagine  that  it  is  Graduation  Day,  After  the  graduates  throw  their  hats  in  the  air,  you  meet  up  with  some 
of  the  young  men  and  women  who  were  cadets  in  your  department's  academic  major  or  minor.  What 
characteristics  would  you  like  them  to  have  as  a  result  of  being  in  your  program?  What  will  they  know? 
What  skills  will  they  have?  What  attitudes  or  habits  of  mind  will  they  possess?  By  imagining  the 
characteristics  that  you  want  cadets  to  have  at  the  end  of  your  academic  major  or  minor,  you  have 
taken  the  first  step  toward  defining  the  learning  outcomes  of  your  program. 

Cadet  learning  outcomes  refer  to  the  knowledge,  skills,  and  attitudes  that  your  cadets  will  possess  at  the 
time  of  graduation  as  a  result  of  participating  in  your  academic  major  or  minor.  Articulating  these 
outcomes  is  a  necessary  first  step  in  developing  the  assessment  program  for  your  major  or  minor,  as 
they  establish  the  "targets"  that  you  hope  to  achieve  and  that  you  use  to  measure  the  extent  of  your 
success.  Therefore,  this  chapter  is  written  to  help  you  write  (or  refine)  the  cadet  learning  outcomes  for 
your  major  or  minor 


A  Word  About  Terminology 

One  of  the  biggest  sources  of  confusion  in  the  assessment  world  involves  the  varied 
language  that  people  use  to  describe  those  things  that  they  want  students  to  learn. 
Sometimes  (as  here),  you  will  hear  people  refer  to  the  "outcomes"  of  an  academic 
program.  Other  times,  you  may  hear  people  use  words  like  "goals"  or  "objectives"  instead. 
Unfortunately,  there  doesn't  appear  to  be  universal  agreement  about  the  distinctions 
between  these  terms,  so  many  people  end  up  using  them  somewhat  interchangeably.  For 
the  purposes  of  this  Handbook ,  be  aware  that  I  will  be  using  the  phrase  "cadet  learning 
outcomes"  to  describe  the  knowledge,  skills,  and  attitudes  that  your  cadets  will  possess  at 
the  time  of  graduation  as  a  result  of  participating  in  your  academic  major  or  minor. 


This  chapter  is  divided  into  two  parts.  The  first  part  reviews  the  characteristics  of  effective  cadet 
learning  outcomes.  Consider  these  characteristics  to  be  "best  practices,"  and  I  encourage  you  to  write 
your  program's  cadet  learning  outcomes  in  ways  that  are  consistent  with  them.  The  second  part  of  the 
chapter  includes  some  more  specific  advice  about  steps  that  you  can  take  if  you  are  working  on 
developing  cadet  learning  outcomes  for  your  academic  major  or  minor. 


18 


Characteristics  of  Effective  Cadet  Learning  Outcomes 

1 *  Effective  cadet  learning  outcomes  articulate  the  knowledge ,  skills ,  and  attitudes  that  cadets  will 
have  at  the  end  of  your  academic  major  or  minor.  The  focus  of  well-written  learning  outcomes  is 
on  what  cadets  will  be  like.  This  is  distinct  from  what  you  -  the  faculty-  will  do  to  help  them  get 
there. 

While  this  seems  like  a  simple  idea,  it  is  quite  common  for  faculty  to  write  cadet  learning 
outcomes  in  ways  that  violate  this  principle.  The  most  common  mistake  occurs  when  faculty 
write  outcome  statements  that  focus  on  what  they  (i.e.,  the  faculty  members)  will  do  in  the 
major  or  minor,  rather  than  what  the  cadets  will  learn  as  a  result  of  being  in  the  program.  For 
instance,  faculty  may  write  that  the  academic  major  will  "cover"  certain  fundamental  knowledge 
in  the  discipline.  This  is  a  mistake,  as  it  places  the  emphasis  on  what  the  faculty  members  will 
teach,  rather  than  on  what  the  cadets  will  learn. 

In  order  to  avoid  this  mistake,  assessment  professionals  typically  write  learning  outcome 
statements  beginning  with  a  phrase  like  the  following: 

"As  a  result  of  being  in  this  academic  major,  cadets  will  ..." 

Notice  that  this  outcome  statement  explicitly  focuses  on  cadets  -  what  they  will  have  achieved 
as  a  result  of  being  in  the  academic  major. 

2.  Effective  cadet  learning  outcomes  are  actionable. 

Well-written  cadet  learning  outcome  statements  are  useful  because  they  guide  action,  both  by 
faculty  and  cadets.  Here  are  three  ways  that  your  outcome  statements  will  play  a  valuable  role 
within  your  academic  major  or  minor. 

®  As  you  will  see,  well-written  cadet  learning  outcome  statements  serve  as  the  foundation 
of  your  assessment  efforts.  Faculty  engage  in  assessment  in  order  to  determine  the 
extent  to  which  cadets  are  learning  what  you  want  them  to  learn.  As  a  result,  your 
program's  assessment  efforts  will  be  effective  to  the  extent  that  they  inform  you  and 
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your  colleagues  how  cadets  are  progressing  toward  accomplishment  of  your  agreed- 
upon  learning  outcomes. 

•  Second,  cadet  learning  outcome  statements  provide  a  common  target  for  faculty. 
Generally  speaking,  cadets  are  not  likely  to  achieve  the  outcomes  set  out  for  them  by 
accident.  Instead,  outcome  achievement  will  come  from  intentional  effort  on  the  part  of 
the  faculty  and  staff  charged  with  designing  and  teaching  courses  within  your  major  or 
minor.  Getting  faculty  within  the  program  to  agree  on  what  those  cadet  learning 
outcomes  are  is  the  first  step  toward  achieving  them. 

•  Finally,  learning  outcome  statements  provide  a  common  target  for  the  cadets  in  your 
academic  major  or  minor.  Cadets  are  busy  people,  and  it  is  not  uncommon  for  them  to 
bounce  from  one  activity  to  the  next  without  giving  too  much  thought  to  why  those 
activities  exist.  If  we  don't  communicate  with  them  about  the  outcomes  we  intend  to 
develop  through  our  activities,  they  are  not  likely  to  put  the  pieces  together  by 
themselves.  Being  transparent  with  your  intentions  can  guide  their  behavior  and  make  it 
much  easier  for  them  to  take  responsibility  for  their  own  development. 

In  order  to  maximize  how  actionable  your  cadet  learning  outcomes  are,  !  encourage  you  to  be 
mindful  of  how  many  you  create  for  your  academic  program.  If  your  major  or  minor  has  only 
one  or  two  cadet  learning  outcomes,  those  outcomes  will  likely  be  written  so  broadly  that  they 
won't  guide  anyone's  actions.  On  the  other  hand,  if  your  program  has  too  many  cadet  learning 
outcomes  (say,  a  dozen  or  more),  faculty  and  cadets  may  feel  pulled  in  so  many  different 
directions  that  they  don't  meaningfully  accomplish  any  of  them.  While  there  is  no  magic  number 
of  cadet  learning  outcomes  to  strive  for,  our  most  effective  academic  programs  have  generally 
written  somewhere  between  3  and  11. 

Another  way  you  can  ensure  that  your  cadet  learning  outcomes  are  actionable  is  to  pay  special 
attention  to  how  you  word  them.  For  instance,  many  programs  have  written  cadet  learning 
outcomes  to  include  action  verbs  that  specify  exactly  what  they  expect  cadets  to  be  able  to  do 
in  order  to  demonstrate  accomplishment  of  them.  So,  you  might  indicate  that  you  want  cadets 
to  "design  experiments,"  or  "communicate  economic  analyses,"  or  "evaluate  historical  data." 
Using  this  approach  refines  the  preferred  format,  such  that  each  outcome  statement  will  look 
more  like  the  following. 
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"As  a  result  of  being  in  this  academic  major,  cadets  wilt  <actlon  verb>  <direct  object>” 

Bloom's  (1956)  Taxonomy,  as  well  as  more  recent  variations  (e.g.,  Anderson  &  Krathwohl,  2001; 
Fink,  2003)  provide  numerous  examples  of  action  verbs  that  can  be  used  to  help  you  specify  the 
cadet  learning  outcome  statements  in  your  academic  major  or  minor. 

3.  Effective  cadet  learning  outcomes  are  clear  and  understandable  to  faculty  and  cadets. 

If  you  really  intend  your  outcome  statements  to  guide  the  behavior  of  faculty  and  cadets  in  your 
academic  major  or  minor,  they  will  need  to  be  written  such  that  both  faculty  and  cadets  know 
what  those  statements  mean,  in  addition,  faculty  and  cadets'  understanding  needs  to  be  shared 
so  that  it  is  clear  what  the  expectations  are  for  cadet  learning  in  your  program.  This  will  make  it 
markedly  easier  for  both  faculty  and  cadets  to  know  how  to  make  progress  towards  the 
outcomes,  as  well  as  to  recognize  when  they  have  actually  been  achieved.  In  general,  everyone 
is  more  likely  to  hit  the  target  if  they  have  a  dear  understanding  of  what  that  target  is. 

In  some  cases,  the  need  for  your  outcomes  to  be  clear  and  understandable  may  force  faculty 
members  to  "unpack"  the  more  complex  outcomes  in  your  major.  For  instance,  many  programs 
have  indicated  that  they  want  their  cadets  to  "think  critically."  By  itself,  however,  it  isn't 
immediately  clear  what  "think  critically"  really  means,  and  different  programs  may  emphasize 
slightly  different  aspects  of  this  important  skill.  Therefore,  to  reduce  confusion,  you  may  find  it 
necessary  (or  at  least  beneficial)  to  break  the  larger  concept  of  "think  critically"  into  a  series  of 
statements  that  delineate  exactly  what  you  mean.  If,  by  critical  thinking,  you  mean  "evaluate 
evidence,"  "construct  arguments,"  or  "make  a  decision  in  the  face  of  competing  perspectives," 
now  is  the  time  to  state  those  more  specific  expectations  for  all  to  see. 

4.  Effective  cadet  learning  outcomes  are  developed  across  your  entire  curriculum . 

As  you  write  cadet  learning  outcomes  for  your  academic  major  or  minor,  keep  in  mind  that  they 
represent  the  knowledge,  skills,  and  attitudes  that  cadets  will  develop  in  your  program  as  a 
whole.  Necessarily,  that  means  that  the  cadet  learning  outcomes  will  not  be  accomplished  in  a 
single  course,  but  will  instead  bridge  across  multiple  courses.  As  such,  your  cadet  learning 
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outcomes  need  to  be  more  than  just  a  collection  of  specific  course  objectives.  Instead,  they  will 
be  connective,  tying  together  the  various  experiences  that  cadets  will  be  having  as  they 
complete  the  requirements  of  your  academic  major  or  minor. 

Taken  together,  these  four  characteristics  describe  what  well-written  cadet  learning  outcome 
statements  look  like,  and  provide  a  guide  as  you  write  (or  refine)  the  outcomes  for  your  own  academic 
major  or  minor.  To  gauge  your  progress,  I  also  refer  you  to  the  first  dimension  ("Cadet  Learning 
Outcomes")  of  the  Academic  Program  Self-Assessment  that  we  have  been  using  within  DF  over  the  past 
several  years.  That  first  dimension  is  shown  in  Table  1  at  the  end  of  this  chapter  (see  p,  24).  The  rubric  is 
shown  in  its  entirety  in  Appendix  A. 

Generating  Cadet  Learning  Outcomes 

Writing  effective  cadet  outcome  statements  can  be  hard  work.  Thankfully,  you  are  not  alone  in  taking  on 
the  task,  and  you  have  access  to  many  other  people  who  can  help  you.  This  section  includes  a  sampling 
of  the  people  you  can  turn  to  for  help  /  guidance  if  you  find  yourself  in  need  of  it. 

1.  Your  Faculty  Colleagues 

Articulating  the  cadet  learning  outcomes  for  your  major  or  minor  is  a  deeply  reflective  exercise, 
and  the  learning  outcomes  you  ultimately  select  will  provide  a  word  picture  of  the 
characteristics  you  want  your  cadets  to  have  upon  graduation.  Given  that  your  entire  program 
will  be  involved  in  achieving  these  outcomes,  it  is  a  good  idea  to  engage  as  many  faculty  within 
your  program  as  you  can  in  generating  them.  This  will  maximize  the  chances  that  your  cadet 
learning  outcomes  will  be  complete,  and  it  will  also  lead  to  greater  buy-in  from  your  faculty 
colleagues. 

Flere  are  some  specific  ideas  for  how  you  can  get  your  departmental  conversations  started- 

a.  Start  with  an  exercise  similar  to  the  one  that  opened  this  chapter  (which  was  based 
on  an  exercise  in  Fink,  2003).  That  is,  ask  your  colleagues  to  imagine  your  cadets 
upon  graduation  and  describe  the  characteristics  they  would  like  cadets  to  have  at 
that  time.  As  1  asked  you,  encourage  your  colleagues  to  think  about  the  knowledge, 
skills,  and  attitudes  that  your  graduates  would  ideally  possess.  How  do  your 
colleagues  want  your  cadets  to  be  different  than  they  were  when  they  started  your 
academic  major  or  minor? 
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b.  Ask  your  colleagues  to  reflect  upon  their  own  professional  experience.  As  Huba  & 
Freed  (2000)  point  out,  learning  outcomes  ought  to  reflect  the  knowledge,  skills, 
and  attitudes  needed  by  professionals  in  your  field.  With  that  in  mind,  encourage 
your  colleagues  to  think  about  what  they  need  in  order  to  be  successful.  This  can  be 
an  especially  powerful  exercise  here  at  USAFA  because  of  our  unique  blend  of 
military  officers  and  civilian  academic  professionals.  By  combining  the  insights  of 
these  two  groups,  you  can  begin  to  piece  together  a  robust  description  of  the 
characteristics  your  cadets  need  to  have  as  they  walk  across  the  graduation  stage, 

c.  Ask  your  faculty  colleagues  to  reflect  on  the  role  your  discipline  can  play  in  the 
accomplishment  of  USAFA' s  larger  institutional  outcomes.  Your  academic  major  is 
not  a  stand-alone  entity;  instead,  it  is  embedded  within  the  larger  context  of  the  Air 
Force  Academy.  Therefore,  the  institution  may  look  to  your  program  to  contribute 
to  the  overall  development  of  skills  like  written  communication,  cultural 
competence,  or  scientific  reasoning.  What  contributions  can  your  department  make 
with  regard  to  these  broader  Academy  outcomes? 

d.  You  might  ask  members  of  your  department  to  complete  a  written  instrument,  such 
as  the  Teaching  Goats  Inventory  (Angelo  &  Cross,  1993).  The  Teaching  Goals 
Inventory  is  essentially  a  values  clarification  exercise,  where  faculty  members  rate  a 
series  of  plausible  learning  outcomes  based  on  how  frequently  they  try  to  achieve 
them.  For  instance,  faculty  in  your  program  may  consider  the  outcome  of  "Develop 
ability  to  distinguish  between  fact  and  opinion"  to  be  absolutely  essential, 
something  they  nearly  always  try  to  achieve  in  their  courses.  On  the  other  hand, 
they  may  rate  "Develop  aesthetic  appreciations"  as  relatively  unimportant, 
something  they  rarely  try  to  achieve  in  their  courses.  If  that  were  the  case,  then  you 
would  likely  develop  an  outcome  relating  to  cadets1  ability  to  distinguish  between 
fact  and  opinion,  but  you  probably  wouldn't  develop  an  outcome  relating  to 
developing  aesthetic  appreciations, 

2.  Professional  Organizations  and/or  Colleagues  at  Peer  Institutions 

The  Air  Force  Academy  is  certainly  not  alone  in  articulating  and  assessing  student  learning 
outcomes.  Therefore,  in  addition  to  conversations  within  your  program,  it  is  also  possible  to  look 
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elsewhere  in  the  higher  education  community  for  guidance  pertaining  to  your  cadet  learning 
outcomes.  For  instance: 

a*  As  noted  in  Chapter  2,  several  of  USAFA's  academic  majors  are  eligible  for  disciplinary 
accreditation.  For  some  of  those  programs,  the  disciplinary  accrediting  body  has  already 
provided  guidance  regarding  how  to  create  programmatic  learning  outcomes.  One  salient 
example  is  ABET,  which  accredits  USAFA's  engineering  and  computer  science  programs  and 
has  identified  student  learning  outcomes  that  are  common  to  all  of  its  accredited  programs 
at  the  undergraduate  level.  It  should  come  as  no  surprise,  then,  that  all  of  USAFA's  ABET's 
accredited  programs  have  cadet  learning  outcomes  that  look  remarkably  similar  to  one 
another. 

b.  Even  if  your  program  does  not  have  a  disciplinary  accrediting  body,  it  is  quite  possible  that  a 
professional  organization  in  your  discipline  has  provided  suggestions  for  the  kinds  of 
outcomes  that  may  be  appropriate  for  your  major  or  minor.  For  instance,  while  the 
American  Psychological  Association  does  not  offer  accreditation  to  undergraduate 
programs,  they  have  still  published  guidelines  for  what  they  consider  an  appropriate 
undergraduate  psychology  program  to  be  (available  online  at 

http://www.apa  ,org/ed/precollege/about/psvmaior-guideline$-pdf),  Similarly,  the  American 
Association  for  the  Advancement  of  Science  (AAAS)  has  also  published  a  series  of 
recommendations  for  undergraduate  programs  in  biology  (available  at 
http://visionandchanEe.org/files/2011/03/VC-Brochure-V6-3.pdf).  If  you  haven't  done  so 
already,  it  is  worth  looking  to  see  if  a  professional  organization  in  your  discipline  has  also 
published  this  kind  of  information. 

c.  Many  of  your  colleagues  at  peer  institutions  are  likely  to  be  engaged  in  articulating  learning 
outcomes  for  their  students.  As  a  result,  you  can  get  some  great  ideas  by  learning  more 
about  the  work  that  these  colleagues  have  already  done.  North  Carolina  State  University 
hosts  an  outstanding  compilation  of  assessment-relafed  resources  from  colleges  and 
universities  around  the  country  that  may  assist  in  such  an  effort.  It  is  available  at: 
http://www2.acs.nc  s  u. edu/UFA/a  re  h  ives/a  ssmt/resource.htm. 

3.  Your  Graduates  and  their  Supervisors 

The  Air  Force  Academy  is  in  the  fortunate  position  that  all  (or  at  least  nearly  all)  of  our  graduates 
are  hired  by  the  same  company  upon  graduation.  This  means  that,  relatively  speaking,  it  is  fairly 
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easy  for  us  to  reach  out  to  our  graduates  once  they  are  on  active  duty,  and  it  is  similarly  easy  to 
be  in  touch  with  their  work  supervisors.  By  maintaining  connections  with  each  of  these 
constituencies,  you  can  gain  valuable  inputs  into  what  your  graduates  need  (or  need  more  of)  in 
order  to  be  successful  as  officers  on  active  duty. 

One  example  of  this  practice  that  is  already  in  place  here  at  USAFA  is  the  Engineering  Program 
Advisory  Council  (EPAC).  This  group  consists  of  professionals  in  the  field  who  are  familiar  with 
the  most  pressing  current  issues  in  engineering  and  who  also  have  an  opportunity  to  work  with 
our  cadets  upon  graduation.  Every  two  years,  this  group  convenes  at  the  Academy  to  share 
ideas,  alert  our  engineering  programs  to  emerging  trends,  and  provide  feedback  on  our 
graduates'  performance.  Might  it  be  possible  for  you  and  your  department  to  create  something 
similar  to  help  you  articulate  (or  refine)  the  learning  outcomes  for  your  major  or  minor? 

4.  Your  Colleagues  Around  USAFA 

Finally,  I  encourage  you  to  remember  that  you  have  a  rich  set  of  connections  here  at  USAFA.  Our 
Departmenta  I  Assessment  Network  consists  of  at  least  one  assessment  POC  in  each  academic 
department,  and  many  of  them  have  already  written  successful  learning  outcomes  for  their 
programs.  As  such,  I  encourage  you  to  draw  upon  their  wisdom  and  expertise  as  you  craft 
outcomes  within  your  own  program.  To  help  you  do  that,  I  have  compiled  all  of  the  cadet 
learning  outcomes  that  exist  as  of  the  writing  of  this  document.  That  list  of  outcomes  is  provided 
in  Appendix  B.  I  encourage  you  to  peruse  this  list,  taking  a  look  at  the  work  that  your  colleagues 
have  done.  Please  feel  free  to  reach  out  to  one  another  as  you  take  the  important  step  of 
articulating  student  learning  outcomes  for  your  academic  major  or  minor. 

Finally,  please  know  that  your  support  network  also  includes  me  (Steve  Jones,  Director  of 
Academic  Assessment)  and  the  other  faculty  and  staff  who  work  in  the  Center  for  Educational 
Excellence  (CEE).  We  are  committed  to  partnering  with  you  in  whatever  way  we  can  to  enhance 
the  effectiveness  of  your  educational  programs.  Pfease  don’t  hesitate  to  call  upon  us  for 
consulting  or  other  assistance. 
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Table  1 

Academic  Program  Self-Assessment 
Cadet  Learning  Outcomes 

This  table  includes  one  row  of  the  broader  Academic  Program  Self-Assessment  rubric  that  has  been  in  place  within  DF.  (The  rubric  is  printed  in 
its  entirety  in  Appendix  A.}  In  conjunction  with  your  colleagues,  identify  which  description  (i.e.,  Level  0,  1,  2,  or  3)  best  matches  the  current  state 
of  cadet  learning  outcomes  within  your  academic  major  or  minor. 


Elements 

Question(s)  To 
Think  About 

Level  0 

Level  1 

Level  2 

Level  3 

Cadet 

Has  my 

Cadet 

Cadet  learning 

Cadet  learning  outcomes 

i  Cadet  learning  outcomes  have 

Learning 

department 

learning 

outcomes  have  been 

have  been  written  in  terms 

been  written  for  this  major  in  a 

Outcomes 

written  cadet 

outcomes 

written  for  this  major* 

of  what  cadets  will  know  or 

way  that  allows  for  meaningful 

learning  outcomes 

have  not 

However,  they  are  not 

be  able  to  do  as  a  result  of 

assessment. 

for  this  major  in  a 

been  written 

written  in  terms  of 

being  in  the  major. 

1,  Outcomes  are  written  in 

way  that  allows  for 

for  this 

what  students  will 

However,  modest 

terms  of  what  cadets  will 

meaningful 

assessment? 

major* 

know  or  be  able  to  do 
as  a  result  of  being  in 
the  major. 

improvements  are  needed 
because  one  of  the 
following  is  true: 

1.  Outcomes  are  not 
actionable  (i.e.,  they 
don't  include  an  action 
verb). 

2.  Outcomes  are  not  clear 

and  understandable  to 
faculty  and  cadets. 

know  or  be  able  to  do  as  a 
result  of  being  in  the  major* 

2*  Outcomes  are  actionable 
(i.e.,  they  include  action 
verbs). 

3.  Outcomes  are  clear  and 

understandable  to  both 
faculty  and  cadets. 
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Chapter  4  -  Building  a  Curriculum  Map 

In  the  previous  chapter,  the  focus  was  on  developing  learning  outcomes  for  your  academic  major  or 
minor.  Once  those  outcomes  are  in  place,  it  is  time  to  take  the  next  step  in  your  assessment  planning 
process:  building  a  curriculum  map.  A  curriculum  map  is  simply  a  matrix  that  shows  how  courses  in  your 
program  are  aligned  with  each  of  the  learning  outcomes  you  have  identified.  In  other  words,  a 
curriculum  map  will  show  you  in  which  courses  particular  outcomes  are  being  developed  and/or 
assessed. 

A  Simple  Example  of  a  Curriculum  Map 

As  an  example  of  what  a  curriculum  map  might  look  like,  consider  Table  2,  which  shows  a  curriculum 
map  from  a  hypothetical  department  at  USAFA.  This  department  has  identified  4  different  learning 
outcomes  (LO#l  -  LO#4)  for  cadets  in  their  major.  There  are  10  different  courses  in  this  major  (labeled 
Course  #1  -  Course  #10)  that  all  cadets  are  required  to  take,  and  an  "X"  in  a  box  indicates  that  the 
course  in  question  both  spends  significant  time  developing  the  associated  outcome  and  includes  a 
significant  graded  test  or  assignment  that  the  department  can  use  for  assessment,  The  pattern  of  X's 
indicates  which  courses  are  aligned  with  which  learning  outcomes  (e.g.,  reading  the  top  row  of  the  map, 
one  can  see  that  Course  #1  is  aligned  with  both  LO#l  and  l_0#3). 
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Table  2 


A  Sample  Curriculum  Map 


Learning  Outcomes 

Course 

10#1 

LO#2 

LO#3 

LO#4 

Course  #1 

X 

X 

Course  #2 

X 

X 

Course  #3 

Course  #4 

X 

Course  #5 

X 

X 

Course  #6 

X 

X 

Course  #7 

X 

X 

Course  #8 

X 

X 

Course  #9 

X 

X 

X 

Course  #10 

X 

What  Can  a  Curriculum  Map  Tell  Us? 

Building  a  curriculum  map  is  helpful  for  multiple  reasons.  First,  a  curriculum  map  will  inform  members  of 
your  department  where  in  the  curriculum  cadets  are  developing  each  of  your  outcomes.  Generally 
speaking,  the  programmatic  learning  outcomes  you  created  in  Chapter  3  are  unlikely  to  be  developed  to 
an  equal  degree  in  each  of  your  academic  courses,  instead,  some  courses  are  likely  to  contribute  heavily 
to  the  development  of  one  learning  outcome,  while  other  courses  contribute  more  heavily  to  the 
development  of  other  outcomes.  Identifying  the  pattern  of  outcome  development  helps  focus  faculty 
members  on  what  is  expected  of  them  when  they  teach  a  particular  course,  and  it  will  also  help  you  and 
your  faculty  colleagues  have  more  effective  oversight  of  what  your  cadets  are  experiencing  in  your 
program. 

As  illustration,  consider  the  sample  curriculum  map  in  Table  2.  If  a  faculty  member  is  assigned  to  be  the 
course  director  for  Course  #1,  this  map  makes  It  clear  that  the  program  is  relying  on  that  course  to 
develop  and  assess  LO#l  and  LO#3.  So,  while  the  faculty  member  may  have  the  latitude  to  make  some 
adjustments  to  the  course  based  on  his/her  own  experience  and  professional  expertise,  any  adjustments 
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to  the  course  will  need  to  be  done  within  the  constraints  outlined  in  the  curriculum  map.  Certainly,  if 
the  course  director  wished  to  stop  developing  and  assessing  either  LOffl  or  L0ff3,  that  action  should  be 
coordinated  with  other  faculty  in  the  program, 

A  second,  related,  benefit  of  creating  a  curriculum  map  is  that  it  will  help  you  and  your  program  ensure 
that  all  cadets  have  an  adequate  opportunity  to  achieve  each  of  the  cadet  learning  outcomes  as  they 
progress  through  the  program.  As  illustration,  let's  return  again  to  the  sample  curriculum  map  in  Table 
2.  Notice  that  LO#l,  LOff2,  and  LO#3  are  developed  across  multiple  courses  in  the  major.  However,  LOff4 
is  only  aligned  with  Course  #2.  Is  that  enough  emphasis  to  help  cadets  fully  develop  LO#4?  If  not,  the 
department  may  wish  to  refocus  one  or  more  of  the  other  courses  so  that  they  do  a  better  job  of 
achieving  LOff4.  One  candidate  could  potentially  be  Course  #3,  which  is  currently  not  shown  as  being 
aligned  with  any  cadet  learning  outcomes5. 

A  third  reason  for  building  a  curriculum  map  is  assessment.  As  will  be  discussed  in  Chapter  5,  one  of  the 
most  beneficial  sources  of  assessment  data  pertaining  to  cadet  learning  outcomes  is  cadet  performance 
on  tests  and  assignments  embedded  within  the  curriculum.  A  curriculum  map  will  identify  the  courses 
that  are  most  likely  to  have  the  kinds  of  tests  and  assignments  that  can  provide  these  forms  of 
embedded  assessment  data.  When  it  comes  time  to  compile  assessment  information  with  regard  to  a 
particular  cadet  learning  outcome,  a  well-designed  curriculum  map  can  help  you  and  your  colleagues 
know  where  to  look.  As  an  illustration,  look  again  at  our  sample  curriculum  map  in  Table  2,  which 
indicated  that  Courses  ffl,  #4,  #6,  f#7,  and  #9  could  be  potential  sources  of  assessment  data  for  LO#l. 

Finally,  building  a  curriculum  map  can  help  guide  continuous  program  improvement.  For  instance, 
imagine  that  LOffl  refers  to  effective  written  communication.  Further,  imagine  that  you  and  your 
colleagues  discover  evidence  that  cadets  are  performing  well  in  some  aspects  of  their  writing,  but  poorly 
in  others.  To  improve  cadet  writing,  one  sensible  strategy  would  be  to  feed  that  information  back  to 
those  courses  (ffl,  #4,  #6,  #7,  &  #9)  that  are  already  aligned  with  LOffl.  Perhaps  those  courses  could  add 
an  extra  lesson  dedicated  to  the  development  of  the  deficient  writing  skill,  or  perhaps  they  could 
emphasize  the  deficient  skill  in  their  grading  of  written  assignments.  Regardless,  those  specific  courses 


5  In  addition  to  discovering  outcomes  that  may  not  be  adequately  developed,  a  curriculum  map  can  also  help  you 
discover  courses  that  are  not  sufficiently  outcomes-focused.  For  instance,  if  it  is  really  true  that  Course  #f3  does  not 
develop  or  assess  any  of  the  program's  cadet  learning  outcomes,  one  might  begin  to  wonder  why  it  remains  a 
required  part  of  the  department's  curriculum. 
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are  probably  the  best  ones  to  adjust  in  order  to  bolster  cadet  performance  in  the  deficient  areas.  As 
such,  knowing  which  courses  are  aligned  with  each  outcome  will  help  close  the  loop  more  effectively. 

Constructing  Your  Own  Curriculum  Map 

As  you  undertake  the  task  of  creating  a  curriculum  map,  there  are  several  important  ideas  to  keep  in 
mind. 

1.  Get  Faculty  Involved. 

A  curriculum  map  will  have  far-reaching  consequences  for  the  courses  in  your  program,  As  such, 
it  is  a  good  idea  to  solicit  broad  faculty  participation  in  its  creation.  You  could  start,  for  instance, 
by  ensuring  that  your  colleagues  are  familiar  with  your  program's  cadet  learning  outcomes  and 
then  work  with  leaders  of  individual  courses  to  align  those  courses  with  the  outcomes,  Follow- 
on  discussions  could  involve  coordinating  with  faculty  to  ensure  that  all  learning  outcomes  are 
adequately  addressed.  Based  on  the  sample  curriculum  map  above,  for  instance,  it  may  be  wise 
to  speak  with  the  course  director  of  Course  #3  to  see  which  outcome(s)  could  most  reasonably 
be  aligned  with  that  course.  In  addition,  it  might  be  wise  to  encourage  one  or  more  course 
directors  to  more  closely  align  with  the  outcome  (LO#4)  that  is  currently  underrepresented  in 
the  curriculum. 

In  discussing  a  curriculum  map  with  your  colleagues,  recognize  that  an  important  philosophical 
issue  may  emerge.  Generally  speaking,  faculty  members  are  hired  for  their  disciplinary 
expertise,  and  the  courses  they  teach  will  likely  reflect  their  depth  of  knowledge  in  the  field. 
However,  the  idea  of  a  curriculum  map  suggests  that  faculty  members  will  need  to  focus  on 
more  than  just  their  disciplinary  knowledge;  your  program  will  also  rely  on  them  to  develop  and 
assess  the  broader  cadet  learning  outcomes.  Awareness  of  faculty  members'  responsibility  to 
the  larger  cadet  learning  outcomes  will  be  especially  important  if  those  outcomes  involve  skills 
(e.g.,  effective  written  communication}  or  responsibilities  (e.g.,  respect  for  human  dignity)  that 
are  not  tied  directly  to  faculty  members'  disciplinary  expertise6. 


h  A  similar  argument  could  be  made  with  regard  to  the  relationship  between  core  courses  and  the  broader 
institutional  outcomes,  tn  order  for  the  Academy  to  achieve  its  broad  institutional  outcomes,  we  rely  on  a  subset 
of  core  courses  to  develop  and  assess  them,  even  if  they  extend  beyond  a  course's  disciplinary  content. 
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2.  Establish  Clear  Expectations. 

In  creating  your  map,  it  is  important  to  ensure  that  you  and  your  colleagues  -  particularly  those 
in  course  director  roles  -  are  very  clear  about  the  expectations  that  come  with  aligning  a  course 
with  a  particular  outcome.  For  instance,  your  department  may  decide  that  alignment  with  an 
outcome  means  that  (1)  a  significant  portion  of  the  course  be  dedicated  to  developing  that 
outcome  in  cadets;  and  (2)  there  is  at  least  one  significant  graded  event  (i.e.,  test  or  assignment) 
in  the  course  that  the  department  can  use  to  assess  cadet  progress  toward  the  outcome. 
Fulfilling  these  expectations  will  obviously  require  effort  on  the  part  of  course  directors  and 
instructors,  so  it  is  important  to  reach  agreement  on  what  these  expectations  should  be. 

3.  Build  a  System  of  Accountability. 

Regardless  of  the  expectations  adopted  in  your  major  or  minor,  it  is  also  a  good  idea  to  establish 
some  sort  of  accountability  mechanism  to  ensure  that  courses  are  actually  achieving  them.  That 
way,  as  course  directors  and  individual  instructors  change,  you  and  your  colleagues  can  still  be 
assured  that  a  given  course  is  contributing  meaningfully  to  particular  cadet  learning  outcomes. 
One  way  that  programs  can  do  this  is  by  requiring  course  directors  to  submit  a  written 
"assessment  plan”  at  the  beginning  of  the  semester  that  lays  out  how  the  course  will  accomplish 
each  of  its  outcomes.  This  plan  is  often  paired  with  some  sort  of  "course  report,”  submitted  at 
the  end  of  the  semester7.  In  addition  to  describing  what  the  course  did  to  accomplish  the 
outcomes,  such  a  report  could  also  summarize  relevant  outcomes  assessment  data.  In  programs 
that  currently  implement  some  version  of  assessment  plans  and/or  course  reports,  these 
documents  are  typically  reviewed  by  a  curriculum  committee,  the  assessment  director,  and 
other  members  of  the  department's  senior  leadership  to  so  that  there  is  broad  awareness  of 
how  cadets  are  developing  throughout  their  academic  program. 

4.  Ensure  That  All  Cadets  Have  The  Opportunity  to  Achieve  The  Learning  Outcomes, 

Finally,  it  is  a  good  idea  to  ensure  that  all  of  your  cadets  have  the  opportunity  to  achieve  your 
cadet  learning  outcomes,  regardless  of  the  path  they  take  through  your  major  or  minor.  In  some 
programs,  this  is  relatively  easy,  as  all  cadets  take  the  same  set  of  required  majors'  classes.  In 
other  programs,  however,  this  can  be  somewhat  trickier,  simply  because  cadets  have  choices 

7  In  some  departments,  this  course  report  is  a  written  document,  often  written  in  the  form  of  a  Memorandum,  In 
other  departments,  it  is  provided  in  the  form  a  briefing  that  is  given  to  departmental  leadership.  More  information 
about  these  end-of-course  reports  is  provided  in  Chapter  6. 
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about  which  of  several  paths  to  take.  In  these  programs,  faculty  may  choose  to  align  their 
outcomes  only  with  the  subset  of  courses  that  all  cadets  are  required  to  take.  Another  strategy 
is  to  align  outcomes  with  multiple  courses  such  that  cadets  will  encounter  them  equally,  no 
matter  what  choices  they  make.  Regardless,  the  key  is  to  make  your  learning  outcomes 
unavoidable  as  cadets  progress  through  your  curriculum. 

As  you  prepare  and/or  review  the  curriculum  map  for  your  own  program,  I  refer  you  to  the  second 
dimension  ("Curriculum  Map")  of  the  Academic  Program  Self-Assessment  we  have  been  using  within  DR 
That  dimension  is  shown  in  Table  5  on  page  35.  The  rubric  is  shown  in  its  entirety  in  Appendix  A. 


Examples  of  Curriculum  Maps 

There  is  no  single  correct  way  to  construct  a  curriculum  map,  and  departments  within  DF  have  taken  a 
variety  of  approaches.  This  section  includes  a  couple  of  examples  so  you  can  get  a  better  feel  for  the 
range  of  possibilities. 

Among  the  most  straightforward  is  the  map  (shown  in  Table  3  on  page  32)  created  by  the  Department 
of  Law  for  its  Legal  Studies  major.  As  in  the  sample  shown  previously,  the  cadet  learning  outcomes 
(referred  to  as  "program  goals")  for  the  Legal  Studies  major  are  listed  across  the  top  and  the  courses 
offered  as  part  of  the  major  are  shown  down  the  left-hand  side.  In  a  departure  from  the  previous 
example,  however,  note  that  required  courses  are  annotated  with  an  asterisk,  making  it  easy  to  see 
where  ail  cadets  will  encounter  the  outcomes.  The  dots  in  the  matrix  represent  the  relationships 
between  the  courses  and  the  program  goals.  For  reference,  the  program  goals  are  shown  at  the  bottom 
of  the  page. 


Table  3 


A  Curriculum  Map  from  the  Department  of  Law 


25  May  2010 


Alignment  of  Law  Course  Goals 
To  DFL  Program  Goals 

Note:  The  goals  reflected  below  are  from  10  January  2006.  Low  440  (Cyherlaw)  and  Law  466  (Advanced  Topics  m 
LOAC)  are  courses  added  since  then.  The  goals  reflected  for  those  new  courses  are  from  their  predecessor  courses 
fLaw  495  and  Law  499  respectively). 


Course  Name 

Crs  # 

1 1(a) 

Kb) 

2(a) 

2(b) 

3(a) 

3(b) 

4 

Law  for  Air  Force  Officers*  (Core) 

Law  220 

• 

i 

1 

• 

• 

* 

• 

Legal  Research,  Writing,  Advocacy* 

Law  221 

* 

* 

|  * 

Crime  and  Criminal  Justice* 

Law  33 1 

• 

* 

1  4 

j 

Business  Law 

Law  340 

• 

\ 

• 

» 

• 

U.S.  Constitutional  Law* 

Law  351 

* 

• 

1  * 

* 

o 

 1 

• 

[ 

Law  and  Literature 

Law  360 

* 

1  • 

* 

I 

i 

Law  of  Armed  Conflict 

Law  361 

* 

• 

• 

Law  for  Commanders* 

Law  421 

• 

• 

* 

* 

• 

• 

. 

Cyberlaw 

Law  440 

• 

* 

• 

National  Security  Law 

Law  456 

• 

• 

* 

• 

9 

• 

International  Law 

Law  46 1 

• 

• 

• 

Advanced  Topics  m  LOAC 

Law  466 

• 

* 

9 

Legal  Studies  Capstone* 

Law  485 

• 

* 

• 

* 

• 

* 

• 

Special  Topic 

Law  495 

— 

Law  and  Economics 

SS  420 

• 

* 

4 

Negotiations 

SS  483 

• 

• 

I 

4 

*  =  Required  course  for  legal  studio  major 


Legal  Studies  Program  Goals 

] .  Develop  officers  who  can  solve  legal  problems  and  critically  think. 

a.  Apply  FIRAC  method  (Facts,  Issue,  Rule,  Application,  Conclusion)  to  solve  legal  problems 
and  analyze  cases 

b.  Apply  additional  critical  thinking  skills 

2.  Develop  officers  wh  o  c  an  comm  on  icatc  effect  i v  ely . 

a.  Demonstrate  public  speaking  skills 

b.  Demonstrate  writing  skills 

3.  Develop  officers  with  knowledge  of  selected  aspects  of  the  law. 

a.  Demonstrate  knowledge  of  function  of  law  and  due  process  of  law 

b.  Demonstrate  knowledge  of  the  American  legal  system 

4.  Develop  officers  who  can  apply  their  knowledge  and  skills  (from  other  legal  studies  program  goals) 
to  the  m  ilitary  and  relate  them  in  other  contexts. 
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Another,  somewhat  more  complex,  approach  is  provided  by  the  Department  of  Civil  and  Environmental 
Engineering  (DFCE;  see  Table  4).  The  curriculum  map  they  have  created  for  their  Civil  Engineering 
program,  shown  on  page  34,  has  additional  features  not  included  in  the  Legal  Studies  example.  For 
instance,  in  recognition  that  some  courses  are  likely  to  contribute  to  a  particular  outcome  in  a  more 
significant  way  than  others,  this  map  provides  an  indication  of  the  strength  of  the  relationship  between 
each  course  and  the  associated  outcomes.  Those  cells  marked  with  a  "3"  indicate  that  the  course  has  a 
'large  contribution"  to  that  outcome,  those  cells  marked  with  a  "2"  indicate  an  "average  contribution" 
to  that  outcome,  and  those  cells  marked  with  a  "1"  indicate  a  "small  contribution"  to  that  outcome. 

It  is  also  worth  noting  that  some  of  the  cells  are  shaded.  The  shaded  cells  indicate  those  courses  that  will 
provide  embedded  assessment  data  pertaining  to  accomplishment  of  each  outcome.  This  makes  it  clear 
that  it  is  possible  for  a  course  to  make  meaningful  contributions  to  the  development  of  a  given  outcome 
without  necessarily  being  responsible  for  providing  assessment  data  at  the  program  level8.  However, 
note  that  at  least  some  assessment  data  are  available  across  the  curriculum  for  each  of  the  program's 
learning  outcomes. 

A  related  strategy  -  not  shown  in  either  the  Legal  Studies  or  Civil  Engineering  maps  -  would  be  to 
differentiate  between  courses  that  "introduce"  an  outcome  from  those  that  "develop"  it  and  those  that 
"assess"  it.  For  instance,  a  particular  outcome  may  be  "introduced"  in  a  200-leve!  course,  further 
"developed"  in  a  300-level  course,  and  ultimately  "assessed"  in  a  400-level  course.  Using  this  strategy,  it 
would  be  easy  to  imagine  a  curriculum  map  with  labels  such  as  "I",  "D,"  and  "A"  to  differentiate 
between  the  different  roles  each  course  plays  in  the  overall  accomplishment  of  each  outcome. 

Regardless  of  the  particular  approach  your  academic  major  or  minor  takes,  it  is  important  to  create  a 
curriculum  map  so  that  faculty  and  staff  understand  the  relationships  between  the  courses  they  teach 
and  the  program's  cadet  learning  outcomes.  Building  such  a  map  will  help  keep  your  faculty  members 
focused,  help  maintain  oversight  of  how  each  outcome  is  being  developed,  show  you  where  assessment 
data  can  reasonably  be  collected,  and  point  you  to  where  to  turn  when  your  department  works  to 
improve  cadets'  outcome  achievement. 


8  Clearly  differentiating  between  those  courses  that  contribute  to  particular  outcomes  and  those  that  provide 
program-level  assessment  data  can  be  useful  in  managing  the  workload  associated  with  assessing  your  outcomes. 
Even  though  many  courses  are  relied  upon  to  develop  each  outcome,  program-level  assessment  data  may  not 
necessarily  be  collected  in  all  of  them. 
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Table  4 

A  Curriculum  Map  from  the  Department  of  Civil  and  Environmental  Engineering 


Department  of  Civil  and  Environmental  Engineering 
(Updated:  August  2009) 

J  Append  Ll  C-1 J 

PC  Os  and  Supporting  Courses  -  Civil  Engineering  Program 


Program  Curricular  Outcomes  (PCOs) 


Course 

0) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

00) 

(11) 

(12) 

Core  Program 

2 

3 

1 

I 

I 

2 

2 

I 

2 

• 

2 

2 

Math  243/245 

2 

I 

Eugr210 

2 

2 

1 

2 

2 

2 

2 

2 

1 

2 

2 

Civ  Engr  330 

2 

2 

■ 

1 

2 

I 

Civ  Engr  351 

I 

1 

2  - 

1 

1 

1 

2 

1 

1 

Civ  Engr  361 

:  j 

1 

I 

2 

Civ  Engr  362 

* 

■  ■ 

2 

i 

2 

2 

2 

V:: 

2 

2 

Civ  Engr  372 

I3 

2 

1 

2 

1 

2 

Civ  Engr  373 

2 

2 

i 

■  i , 

2 

2 

2  • 

Civ  Engr  390 

r  V-’  ' 

2 

2 

3 

2 

2 

2 

2 

2 

2 

,  Civ  Engr  474 

3 

2 

2 

2 

2 

2 

Civ  Engr  4  SO 

3 

3 

2 

3 

2 

3 

3 

2 

■  3 

2 

3 

3 

Civ  Engr  45  & 

l  "M** 

3  \ 

r: 

3 

2 

2 

2 

2 

E402  (FE) 

3 

3 

2 

2 

3 

Civ  laigr  405 

2 

2 

2 

2 

2 

3 

3 

Design  Opt  I/E 

3 

3 

3 

2 

3 

B as  Sci  Opt 

2 

Engr  Qpl 

2 

Academy  Opt 

. 

1  -  Small  Contribution;  2  -  Average  Contribution;  3  -  Large  Contribution 
Sba<&d  Course  -  from  which  assessment  events  are  selected  to  evaluate  PCO 
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Table  5 

Academic  Program  Self-Assessment 
Curriculum  Map 

This  table  includes  one  row  of  the  broader  Academic  Program  Self-Assessment  rubric  that  has  been  in  place  within  DF.  {The  rubric  is  printed  in 
its  entirety  in  Appendix  A.}  In  conjunction  with  your  colleagues,  identify  which  description  (i.e.,  Level  0, 1,  2,  or  3)  best  matches  the  current  state 
of  cadet  learning  outcomes  within  your  academic  major  or  minor. 


Elements 

Questions)  To 
Think  About 

Level  0 

Level  1 

Level  2 

Level  3 

Curriculum 
Map 
(Note:  A 
"curriculum 
map"  is  a 
matrix  that 

shows  which 
courses  in 
your  major 
or  minor  are 
aligned  with 
which  of 
your  cadet 
learning 
outcomes.) 

Has  my 
department 
developed  a 
curriculum  map 
showing  dear 
linkages  between 
cadet  learning 
outcomes  and  the 
learning 
experiences  (to 
include 

coursework)  in  the 
major? 

No 

curriculum 
map  has 
been 

developed. 

A  curriculum  map  has 
been  developed. 
However,  significant 
improvements  are 
needed  because  of  one 
or  both  of  the  following: 

1.  There  are  one  or 
more  required 
courses  that  do 
not  map  to  any 
outcomes. 

2.  There  are  one  or 

more  outcomes 

that  are  not 
addressed  in  the 
curriculum. 

A  curriculum  map  has  been 
developed,  showing  clear 
linkages  between  most 
cadet  learning  outcomes 
and  the  learning 
experiences  (to  include 
coursework)  provided  in  the 
major.  However,  modest 
improvements  are  needed 
because  one  or  more 
outcomes  lack  adequate 
coverage. 

A  curriculum  map  has  been 
developed,  showing  clear 
linkages  between  all  cadet 
learning  outcomes  and  the 
learning  experiences  (to  include 
coursework)  in  the  major. 

36 


Chapter  5  -  Assessment  Methods:  Measuring  Student  Accomplishment9 

In  previous  chapters,  we  focused  on  articulating  your  programmatic  cadet  learning  outcomes  (Chapter 
3}  and  mapping  your  curriculum  to  those  outcomes  (Chapter  4).  in  this  chapter,  we  turn  our  attention  to 
the  various  methods  that  can  be  used  to  assess  the  extent  to  which  cadets  are  actually  accomplishing 
those  outcomes.  The  journey  that  cadets  take  on  the  way  to  your  learning  outcomes  is  a  iong  one,  and 
assessment  methods  can  inform  you  about  how  they  are  progressing  (see  Figure  2}  and  when  they  make 
it  to  the  desired  destination. 

Figure  2.  The  Role  of  Assessment  in  Monitoring  Progress  toward  Accomplishment  of  your  Outcomes10 


Starting  Point 


Milestone  Milestone  Milestone 


Progress  toward 
Accomplishment  of  Outcome 


Complete  & 
Successful 
Accomplishment 
of  Outcome 


What  does  the  Evidence  Say? 


The  first  section  of  the  chapter  includes  some  general  ideas  that  are  important  to  keep  m  mind  as  you 
consider  the  assessment  methods  that  are  most  appropriate  for  your  academic  major  or  minor.  Then,  in 
the  second  part  of  the  chapter,  I  describe  the  many  different  assessment  methods  that  are  currently 
being  used  around  DF,  to  include  the  strengths  and  weaknesses  of  each  approach.  At  the  end  of  the 
chapter,  there  is  an  up-to-date  summary  of  which  programs  are  using  which  methods,  should  you  wish 
to  draw  upon  the  experience  and  wisdom  of  colleagues  who  have  experience  working  with  each 
method. 


This  chapter  draws  heavily  from  the  American  Psychological  Association's  Assessment  Cyberguide,  available  for 
download  at:  http ://www . apa.o rg/ed/Eovern a n ce/bea/a ssessm ent-cvbemu ide-v2.pdf . 

1C  Thank  you  to  Bob  Noyd  (DFB)  for  his  creation  of  this  figure. 
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Four  Ideas  to  Keep  in  Mind 

1.  Be  Sure  to  Link  Assessments  to  Programmatic  Cadet  Learning  Outcomes.  As  you  review  the 
assessment  methods  described  in  this  chapter,  remember  that  the  purpose  of  these  tools  is  to  provide 
you  and  your  colleagues  with  information  about  the  extent  to  which  cadets  are  achieving  your 
programmatic  learning  outcomes.  If  cadets  are  making  progress  to  your  program's  satisfaction,  you  and 
your  colleagues  can  celebrate.  If  they  are  not,  then  your  task  is  to  gather  specific  enough  information 
about  cadets'  learning  that  the  program  can  make  necessary  adjustments  to  its  processes.  In  short,  the 
assessment  methods  your  program  uses  will  be  beneficial  to  the  extent  that  they  provide  the  program 
with  information  about  cadets'  progress  toward  accomplishment  of  the  outcomes. 

The  idea  of  using  assessment  as  evidence  about  outcome  accomplishment  becomes  especially 
important  as  you  consider  assessment  methods  that  are  embedded  within  academic  courses.  As  an 
example,  many  faculty  administer  exams,  assign  papers,  or  ask  cadets  to  give  oral  presentations  in  their 
classes,  and  these  methods  can  be  outstanding  ways  to  gather  information  about  cadets'  progress 
toward  accomplishment  of  programmatic  learning  outcomes.  However,  the  mere  presence  of  these 
graded  events  is  not  enough.  To  inform  your  department  about  outcome  achievement,  those  graded 
events  (or  at  least  parts  of  those  graded  events)  need  to  be  deliberately  targeted  toward  specific 
programmatic  cadet  learning  outcomes,  and  the  evidence  they  generate  needs  to  be  reviewed  and  used 
beyond  the  course,  at  the  program  level. 

2.  Direct  vs.  Indirect  Measures  of  Learning.  This  chapter  describes  both  direct  and  indirect  measures  of 
cadet  learning.  Direct  measures  are  those  that  require  cadets  (or  graduates}  to  display  their  progress 
toward  accomplishment  of  your  learning  outcomes.  These  include  course-embedded  tests  and 
assignments,  summative  measures  of  cadet  performance  (such  as  standardized  tests),  and  examinations 
of  how  graduates  perform  on  active  duty.  In  contrast,  indirect  measures  of  cadet  learning  are  based  on 
cadets'  (or  graduates')  reflections  on  their  learning,  rather  than  on  an  actual  display  of  that  learning. 
Indirect  measures  include  cadet  journals,  surveys,  and  focus  groups. 

As  you  will  see,  both  direct  and  indirect  measures  can  be  quite  informative.  However,  it  is  best  to  build 
at  least  some  direct  measures  into  your  assessment  program.  Because  direct  measures  require  cadets 
(or  graduates)  to  demonstrate  what  they  have  learned,  they  provide  more  compelling  evidence  than 
indirect  measures,  which  cadets  could  potentially  "fake"  or  provide  misleading  information  on.  As  you 
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review  the  strategies  implemented  across  DF,  you  will  find  that  the  most  robust  assessment  programs 
are  those  that  rely  heavily  on  direct  measures  and  use  indirect  measures  as  helpful  supplements. 

The  distinction  between  direct  and  indirect  measures  of  learning  is  emphasized  in  the  Assessment 
Methods  dimension  of  the  Academic  Program  Self-Assessment  we  have  been  using  within  DF.  That 
dimension  is  shown  in  Table  6  on  page  48.  The  Self-Assessment  is  shown  in  its  entirety  in  Appendix  A. 

3 ,  Quantitative  vs.  Qualitative  Methods .  As  you  learn  about  the  various  assessment  methods  in  this 
chapter,  you  will  see  that  some  methods  are  quantitative  in  nature,  while  others  are  qualitative  in 
nature.  For  some,  the  apparent  precision  of  quantitative  measures  {which  yield  numerical  results)  may 
appear  superior  to  qualitative  measures  (which  do  not).  Be  aware,  however,  that  there  may  be 
situations  in  which  the  richness  of  qualitative  data  may  be  far  more  useful  than  a  {potentially  less 
meaningful)  number.  Again,  combining  results  from  each  kind  of  measure  is  likely  to  lead  to  the  best 
possible  results, 

4.  In  General There  is  No  "Best"  Method .  As  the  discussion  above  suggests,  there  is  no  single  "best" 
method  for  you  to  use  in  assessing  your  cadet  learning  outcomes.  Every  method  has  both  strengths  and 
weaknesses.  As  such,  it  is  wise  to  use  multiple  methods  to  assess  each  of  your  programmatic  learning 
outcomes.  By  itself,  each  method  may  be  imperfect;  taken  together,  multiple  methods  have  a  much 
better  chance  of  providing  converging  evidence  about  the  extent  to  which  your  learning  outcomes  are 
being  successfully  achieved. 


39 


Descriptions  of  Assessment  Methods 

This  section  introduces  a  variety  of  different  assessment  methods.  For  simplicity,  these  methods  are 
organized  into  four  distinct  categories.  The  first  three  categories  {Course-Embedded  Tests  and 
Assignments,  Summative  Performance  Assessment,  and  Examination  of  Graduate  Performance)  are  all 
examples  of  direct  assessment  measures.  The  fourth  category  deals  specifically  with  indirect  measures. 
This  categorization  scheme  is  represented  visually  in  the  concept  map  below  (Figure  3). 


Figure  3.  A  Concept  Map  of  Assessment  Methods 
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Direct  Measures  of  Cadet  Learning 

1.  Course-embedded  tests  and  assignments  -  As  the  name  suggests,  these  are  tests  and 

assignments  (or  even  portions  of  tests  and  assignments)  that  are  embedded  within  the  normal 
flow  of  academic  courses.  In  addition  to  receiving  a  grade,  cadets'  work  is  also  evaluated  in 
terms  of  outcome  achievement. 


Method 

Strengths 

Weaknesses 

A.  Objective  Exams 
(e.g.,  multiple 
choice,  true/false, 
fill-in-the-blank 
items) 

•  Easy  to  embed  within  many 
existing  courses 

•  Faculty  can  score  them  quickly, 
easily,  and  reliably 

•  "Benchmark"  items  can  be  used 
across  multiple  semesters  to 
gauge  changes  in  cadet 
performance 

•  Often  tests  low-level  knowledge; 
may  sacrifice  depth  for  breadth 

•  Constructing  high-quality  test 
questions  is  more  difficult  than 
it  may  initially  appear 

•  "Benchmark"  items  are 
compromised  if  cadets  gain 
access  to  them 

B.  Workout  Problems  / 
Problem  Sets 

•  Easy  to  embed  within  many 
existing  courses 

•  Can  be  scored  reliably  if  an 
effective  scoring  rubric  is  used 

•  Especially  helpful  for  technical 
disciplines 

•  Cadets  and  faculty  may  focus 
more  on  arriving  at  the  correct 
answer,  rather  than  on  the 
thought  process  that  led  to  that 
answer 

•  Cadets  may  learn  to  "pattern 
match,"  allowing  them  to 
complete  problems  without 
deep  understanding  of  course 
content 

C.  Essay  Exams 

•  Easy  to  embed  within  many 
existing  courses 

•  Can  demonstrate  higher-level 
learning,  to  include  application, 
synthesis,  and  critical  thinking 
skills 

•  Can  be  scored  reliably  if  an 
effective  scoring  rubric  is  used 

•  Without  an  effective  rubric, 
scoring  may  be  unreliable. 

•  Scoring  can  be  time  and  energy- 
intensive 

•  Essay  questions  are  likely  to 
focus  on  one  aspect  of  outcome 
performance;  may  sacrifice 
breadth  for  depth 

D.  Writing  Assignments 
(e.g.,  term  papers, 
lab  reports,  etc.) 

•  Easy  to  embed  within  many 
existing  courses 

•  Forces  cadets  to  articulate  their 
understanding  of  course 
material 

•  Helps  cadets  develop  valuable 
writing  skills 

•  Without  an  effective  rubric, 
scoring  may  be  unreliable 

•  Scoring  can  be  time  and  energy- 
intensive 

•  Scores  may  conflate  subject 
matter  knowledge  with  writing 
skill.  For  the  purposes  of 
outcomes  assessment,  it  is 
important  to  treat  these 
separately 
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Method 

Strengths 

Weaknesses 

E.  Oral  Presentations 
(e.g.,  speech, 
debate,  etc.) 

•  Easy  to  embed  within  many 
existing  courses 

•  Forces  cadets  to  articulate  their 
understanding  of  course 
material 

•  Helps  cadets  develop  valuable 
speaking  &  presentation  skills 

•  Authentic  tasks  will  generally 
motivate  cadets  to  do  their  best 
work 

»  Without  an  effective  rubric, 
scoring  may  be  unreliable. 

•  Scoring  can  be  difficult, 
especially  if  the  presentation 
isn't  recorded.  Graders  are 
unable  to  view  the  presentation 
more  than  once 

•  Scores  may  conflate  subject 
matter  knowledge  with  speaking 
/  presentation  skills.  For  the 
purposes  of  outcomes 
assessment,  it  is  important  to 
treat  these  separately 

F.  Poster  Presentations 
(i.e.,  similar  to  those 
at  professional 
conferences) 

•  Easy  to  embed  within  many 
existing  courses 

•  Provides  opportunity  for  cadets 
to  communicate  in  multiple 
ways  (e.g.,  writing,  graphically, 
orally) 

•  Allows  for  expert  review  / 
judgment;  can  be  facilitated 
with  clear  scoring  criteria 

•  Can  be  easier  to  grade  than 
longer  papers  /  assignments 

•  Authentic  tasks  will  generally 
motivate  cadets  to  do  their  best 

work 

•  Without  an  effective  rubric, 
scoring  may  be  unreliable 

•  Scoring  can  be  difficult, 
particularly  if  the  oral 
presentation  is  one  of  the  things 
being  measured  (see  also  "Oral 
Presentations") 

•  Scores  may  conflate  aesthetic 
sense,  social  interaction  skills, 
and  subject  matter  expertise. 

For  the  purposes  of  outcomes 
assessment,  it  is  important  to 
treat  these  separately 

G.  Other  Embedded 
Cadet  Performances 
(e.g.,  design 
projects,  computer 
programs,  etc.) 

*  Easy  to  embed  within  many 
existing  courses 

*  Assesses  complex  skills  needed 
by  professionals 

•  Allows  for  expert  review  / 
judgment;  can  be  facilitated 
with  dear  scoring  criteria 

•  Authentic  tasks  will  generally 
motivate  cadets  to  do  their  best 
work 

•  Complex  performances  will  likely 
require  a  great  deal  of 
scaffolding  in  class  in  order  to 
promote  cadet  success.  Cadets 
won't  necessarily  know  how  to 
be  successful  unless  they  receive 
substantial  guidance 

•  Without  an  effective  rubric, 
scoring  may  be  unreliable 

•  Overall  scores  may  be  a  function 
of  multiple  outcomes.  For  the 
purposes  of  outcomes 
assessment,  it  is  important  to 
treat  these  separately 

H.  Performance  in 
Courses  Outside 

Your  Program 

•  Provides  information  for 
outcomes  not  specifically 
addressed  in  your  curriculum 

•  Fosters  cross-talk  between 
academic  departments 

*  Requires  coordination  with 
other  departments 

•  Grades  (for  example)  are  a  crude 
measure  of  outcome 
achievement 
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2.  Summative  Performance  Assessment  -  The  methods  in  this  category  tend  to  be  implemented  / 
reviewed  at  the  end  of  a  program,  perhaps  shortly  before  graduation.  Cadets'  performance  on 
these  summative  assessments  reflects  upon  the  success  /  failure  of  the  program  as  a  whole. 


Method 

Strengths 

Weaknesses 

A.  Standardized  Tests 
(e.g.,  FE  Exam, 

Major  Field  Test) 

•  Allows  for  comparisons  of 
cadet  performance  with 
national  norms 

•  Allows  for  trend  analysis, 
examining  patterns  of  cadet 
performance  on  the  same 
instrument  over  time 

•  Scoring  burden  is  typically 
offloaded  to  testing  company 

»  Existing  tests  may  not  match 
the  specific  learning  outcomes 
in  your  program 

•  Comparisons  with  national 
norms  can  be  misleading, 
particularly  if  you  don't  know 
the  conditions  under  which 

students  at  other  institutions 

are  tested 

•  Potentially  high  financial  cost 

•  Potentially  low  cadet  buy-in, 
leading  to  poor  motivation  to 
do  well 

•  Potentially  low  faculty  buy-in, 
leading  to  an  unwillingness  to 
make  adjustments  based  on 
test  results 

•  Feedback  on  results  is 
potentially  slow 

B.  Locally-developed 
summative  exams 
(e.g.,  a  USAFA- 
developed  exam 
administered  to  all 
first-class  cadets  in 
an  academic  course) 

•  Can  be  tailored  to  match 
specific  learning  outcomes  of 
your  program 

•  Allows  for  trend  analysis, 
examining  patterns  of  cadet 
performance  on  the  same 
instrument  over  time 

•  Relatively  low  financial  cost 

•  Allows  for  quicker  grading  and 
feedback  than  nationally- 
standardized  tests 

*  Test  creation  may  be  complex 
and  time  consuming 

*  Because  exam  is  developed 
locally,  national  norms  are  not 
available 

*  Potentially  significant  cost  in 
terms  of  time  dedicated  to 
grading 

®  Test  is  compromised  if  cadets 
gain  access  to  it 
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Method 

Strengths 

Weaknesses 

C.  Capstone 

experiences  (e>g<, 
research  projects, 
internships, 
competitive  teams) 

•  Authentic  tasks  will  generally 
motivate  cadets  to  do  their 

best  work 

♦  Aliows  for  expert  review  / 
judgment;  can  be  facilitated 
with  dear  scoring  criteria 

•  Potentially  expensive  in  terms 
of  both  time  and  money 

•  Unless  required  in  your 
program,  on  ly  some  of  your 
cadets  (often  only  the  best 
ones)  will  participate.  This  limits 
what  your  department  can 
conclude  about  the  learning  of 
all  cadets  in  your  program 

•  Overall  performance  may  be  a 
function  of  multiple  outcomes. 
For  the  purpose  of  outcomes 
assessment,  it  is  important  to 
treat  these  separately 

D.  Cadet  Portfolios 

®  8y  examining  changes  in 
cadets'  work  over  time,  it  is 
possible  to  see  growth  for  each 
individual  cadet 

•  Helpful  for  identifying  strengths 
and  weaknesses  of  individual 
cadets;  allows  for  remediation, 
if  necessary 

•  Encourages  cross-talk  between 
courses  regarding  what  is 
appropriate  for  cadets  to 
include  in  their  portfolios 

•  Cadets  are  allowed  to  highlight 
and  reflect  upon  their  best 
work 

*  Without  significant  reflection  by 
cadets,  a  portfolio  may  become 
nothing  more  than  a  collection 
of  artifacts 

*  Time  consuming  and 
challenging  to  evaluate 

*  Without  significant  oversight, 
the  content  of  cadets' 
portfolios  may  vary  a  great  deal  1 
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3.  Examination  of  Graduate  Performance  -  Given  that  (nearly)  all  of  our  cadets  go  on  to  work  for 
the  same  employer,  it  is  relatively  easy  for  us  to  gather  information  about  how  they  perform 
when  they  are  on  active  duty. 


Method 

Strengths 

Weaknesses 

A.  Archival  Data  (e.g., 
review  of  promotion 
rates,  OPR's, 
graduate  school 
performance,  etc.) 

•  A  great  deal  of  data  already 
exists  (e.g.,  in  the  Air  Force 
personnel  system) 

•  Information  is  provided  by 
external  stakeholders  who  can 
provide  a  fresh  perspective  on 
graduates'  performance 

•  Review  of  these  data  focuses 
faculty  attention  on  the  quality 
of  graduates  we  ultimately 
produce 

•  While  data  do  exist,  gaining 
access  to  them  can  be 
challenging 

•  Existing  data  may  not  speak 
specifically  to  cadet 
achievement  of  your  learning 
outcomes 

•  Graduate  performance  depends 
on  many  factors  besides  the 
quality  of  your  program, 
especially  as  the  time  between 
graduation  and  data  collection 
increases 

B.  External  Stakeholder 
Feedback  (e.g.. 
Engineering 

Program  Advisory 
Council,  etc.) 

•  External  stakeholders  can 
provide  a  fresh  perspective  on 
graduates'  performance 

•  Review  of  these  data  focuses 
faculty  attention  on  the  quality 
of  graduates  we  ultimately 
produce 

•  Allows  for  validation  of 
programmatic  outcomes  to 
ensure  that  the  program  is 
focused  on  those  things  that 
the  Air  Force  values 

•  Data  may  be  difficult  and/or 
expensive  to  gather 

•  Information  is  likely  to  come 
from  a  relatively  small  number 
of  people  and  may  not  be 
representative  of  the  larger 
population 

•  Validating  that  your  outcomes 
are  appropriate  says  nothing 
about  how  well  cadets  are 
achieving  those  outcomes 

C,  Supervisor 

Interviews  /  Surveys 

•  Information  is  provided  by 
external  stakeholders  who  can 
provide  a  fresh  perspective  on 
graduates'  performance 

•  Review  of  these  data  focuses 
faculty  attention  on  the  quality 
of  graduates  we  ultimately 
produce 

•  Surveys  can  be  targeted  to 
assess  graduates'  ability  to 
demonstrate  your  learning 
outcomes  on  the  job 

•  Nearly  all  of  our  graduates 
work  for  the  same  employer, 
making  outreach  to  supervisors 
relatively  straightforward 

•  Difficult  to  achieve  high 
response  rate,  potentially 
compromising  value  of  the 
method 

•  Some  respondents  may  be 
motivated  not  to  provide 
honest  feedback  (e.g.,  because 
they  don't  want  to  be  the 
bearer  of  bad  news) 
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Indirect  Measures  of  Cadet  Learning 

These  strategies  rely  on  asking  cadets  {or  graduates}  to  reflect  upon  their  USAFA  experience  and  what 
they  have  learned  from  it.  In  general,  the  data  are  relatively  easy  to  collect,  but  cadets'  responses  may 
not  necessarily  reflect  their  actual  accomplishment  of  the  learning  outcomes. 


Method 

Strengths 

Weaknesses 

A-  Cadet  Journals  / 
Reflections 

•  Easy  to  embed  within  existing 
courses 

»  Encourages  cadet  reflection  on 
what  they  have  accomplished 
and/or  how  they  have 
developed  in  your  program 

•  Cadets'  responses  may  not 
accurately  reflect  their  actual 
accomplishment  or  how  they 
have  actually  developed 

*  Cadets  do  not  necessarily  know 
how  to  judge  their  own  work  or 
development.  Substantial 
assistance  and/or  support  may 
be  needed 

Course  Feedback 

•  Data  are  easy  to  collect  within 
academic  courses 

•  Existing  online  infrastructure 
makes  it  easy  to  customize 
feedback  form  to  ask  about 
programmatic  outcomes 

•  Existing  infrastructure  allows 
for  DF  norms  and  analyses  of 
trends  over  time 

•  Data  can  be  used  to  help 
Individual  faculty  members 
improve  their  teaching 

•  Cadets'  responses  may  not 
accurately  reflect  their  actual 
accomplishment  or  how  they 
have  actually  developed 

•  Without  intentional  effort  (e.g., 
collecting  data  during  class 
time),  response  rates  are  likely 
to  be  low.  Results  can  be 
influenced  by  who  chooses  to 
participate 

C.  Cadet  Interviews/ 
Surveys  (e.g.,  of 
first-class  cadets, 
shortly  before  they 
graduate) 

•  Data  are  easy  to  collect  in 
academic  classes  or  within 
other  dedicated  academic  time 
slots  (e,g,,  Dean's  Time) 

•  Interviews  /  Surveys  can  be 
customized  to  focus  on  specific 
programmatic  outcomes 

•  Can  provide  personal 
perspective  on  cadets1 
experience  in  the  program 

*  Cadets1  responses  may  not 
accurately  reflect  their  actual 
accomplishment  or  how  they 
have  actually  developed 

*  Results  can  be  influenced  by 
who  chooses  to  participate. 

*  Results  can  be  influenced  by 
the  quality  of  the  survey, 
interviewer,  and/or  protocol 

46 


Method 

Strengths 

Weaknesses 

D.  Focus  Groups  / 
Majors'  Meetings 
(e.g.,  with  first-class 
cadets,  shortly 
before  they 
graduate) 

•  Protocol  can  be  customized  to 
focus  on  specific  programmatic 
outcomes 

•  Can  provide  even  more  in- 
depth  perspective  on  cadets' 
experience  in  the  program 

•  May  reveal  previously  unseen 
areas  of  concern 

•  Relatively  time-intensive 

•  Data  will  be  gathered  from  a 
subset  of  the  cadets  in  your 
program.  Results  can  be 
influenced  by  who  participates 

•  Results  can  also  be  influenced 
by  the  quality  of  the  focus 
group  facilitator  and  the 
questions  asked 

E.  Graduate  Interviews 
/  Surveys 

•  Interviews  /  Surveys  (including 
those  administered  by  the 
institution  at  large)  can  be 
customized  to  focus  on  specific 
programmatic  outcomes 

•  Can  provide  personal 
perspective  on  graduates' 
experience  in  the  program 

•  Graduates  may  have  broader 
perspective  on  their  USAFA 
experience  once  they  have 
been  in  the  operational  Air 

Force  for  a  while 

•  Difficult  to  achieve  high 
response  rate,  potentially 
compromising  value  of  the 
method 

•  Graduates'  responses  may  not 
accurately  reflect  their  actual 
accomplishment  or  how  they 
actually  developed 

•  Graduates'  responses  may  be 
influenced  by  numerous  factors 
other  than  the  quality  of  your 
program 

Conclusion 

All  assessment  methods  have  both  strengths  and  weaknesses,  meaning  that  there  is  not  likely  to  be  a 
single  "best"  method  for  use  in  your  program.  Rather  than  looking  for  a  single  "best"  method,  it  may  be 
wiser  to  consider  a  combination  of  methods,  recognizing  that  multiple  methods  are  likely  to  provide  you 
with  richer  information  about  how  well  cadets  are  progressing  toward  accomplishment  of  the  outcomes 
than  any  single  method  would  by  itself.  Adopting  multiple  methods  is  also  wise  because,  as  you  will 
inevitably  see,  some  methods  are  better  matched  to  particular  outcomes  than  others  are.  For  instance, 
objective  exams  can  be  credibly  used  to  assess  cadets'  fundamental  knowledge  in  your  discipline. 
However,  they  would  be  far  less  appropriate  for  assessing  cadets'  attitudes  or  beliefs.  For  these  types  of 
outcomes,  other  methods  are  almost  certainly  more  appropriate. 

If  you  are  considering  the  incorporation  of  a  new  methodology  into  your  academic  assessment  plan,  I 
would  encourage  you  to  move  slowly.  Try  the  new  method  out  on  a  small  scale  and  see  if  it  provides  you 
with  valuable  information.  If  it  does,  then  you  can  consider  expanding  the  scope  of  your  efforts.  If  it 
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doesn't,  then  you  can  either  revise  yourapproaeh  or  abandon  the  method  in  favor  of  something  that 
will  be  more  useful  for  your  particular  needs.  Above  all,  stay  focused  on  the  overarching  purpose  of 
gathering  assessment  data,  which  is  to  inform  you  and  your  colleagues  about  the  degree  to  which 
cadets  are  progressing  toward  successful  accomplishment  of  your  stated  learning  outcomes. 

Another  good  idea  for  adopting  a  new  assessment  methodology  is  to  learn  from  DF  colleagues  who  have 
blazed  the  trail  before  you.  All  of  the  assessment  methodologies  described  in  this  chapter  are  currently 
being  used  somewhere  around  DF.  Rather  than  reinvent  the  wheel,  it  would  be  wise  to  touch  base  with 
those  who  have  experience  using  particular  methods.  They  can  almost  certainly  help  you  make  best  use 
of  any  method  you  are  considering.  To  facilitate  this  type  of  cross-talk  between  programs,  I've  compiled 
Table  7,  shown  on  page  49*  This  table  includes  a  row  for  each  of  the  assessment  methods  discussed  in 
this  chapter  and  a  column  for  each  of  USAFA's  academic  majors.  The  "X's"  in  the  table  indicate  which 
methods  the  assessment  representatives  from  the  relevant  department(s)  report  that  they  currently 
use.  So,  for  instance,  if  you  were  interested  in  learning  more  about  the  use  of  portfolios  at  USAFA,  1 
would  encourage  you  to  seek  out  your  peers  in  English,  Math,  or  Philosophy. 
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Table  6 

Academic  Program  Self-Assessment 
Assessment  Methods 


This  table  includes  one  row  of  the  broader  Academic  Program  Self-Assessment  rubric  that  has  been  in  place  within  DF.  (The  rubric  is  printed  in 
its  entirety  in  Appendix  A.)  In  conjunction  with  your  colleagues,  identify  which  description  (i.e.,  Level  0, 1,  2,  or  3)  best  matches  the  current  state 
of  cadet  learning  outcomes  within  your  academic  major  or  minor. 


Elements 

Questions)  To 
Think  About 

Level  0 

Level  1 

Level  2 

Level  3 

Assessment 

Methods 

Is  my  department 
making  efforts  to 

assess  each  of  the 

cadet  learning 

outcomes  in  the 

major?  To  what 

extent  are  we 

relying  on  direct 

vs.  indirect 

measures? 

No  efforts 

are  being 

made  to 

assess  cadet 

learning 

outcomes. 

_ _ _ 

The  department  is 
beginning  to  assess  at 

least  some  cadet 

learning  outcomes. 
However,  assessment 

efforts  rely  completely 

on  indirect  measures  of 

learning  (e.g.,  cadet 
feedback,  surveys,  focus 
groups,  etc.). 

Assessment  of  cadet 

learning  outcomes  is 
underway  but  not  fully 
implemented.  At  least  one 
of  the  outcomes 
incorporates  the  use  of 
direct  measures  of  learning 
(e.g.,  exams,  papers, 
projects,  portfolios,  etc.). 

Department  is  fully  engaged  in 
assessing  each  of  the  cadet 
learning  outcomes.  Direct 
measures  of  learning  (e.g., 
exams,  papers,  projects, 
portfolios,  etc.)  are  used  and  are 
supplemented  by  indirect 
measures  (e.g.,  cadet  feedback, 
surveys,  focus  groups,  etc.),  as 
appropriate. 
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Table  7:  SUMMARY  Of  ASSESSMENT  METHODS,  BY  ACADEMIC  MAJOR 


Method 

Aero 

Eng 

Ast 

Eng 

6  to 

Befa 

Sci 

Chem 

Civ 

Eng 

Comp 

5d 

Comp 

Eng 

Econ 

Elec 

Eng 

Engl 

Env 

Eng 

FAS 

Geo 

Hist 

Mgt 

Math 

Mech 

Eng 

Met 

MSS 

OR 

Phys 

Phil 

Pol 

Sci 

Space 

Sys 

Eng 

Direct  - 

Embedded 

Objective 

Exam 

Problem  Sets 

Essay  Exam 

Writing 

Oral 

Poster 

Other 

Performance 

Performance 
Outside  Dept 

\ 

*  \ _ 

Direct  - 

Summative 

.  a  i 

\ 

Standardized 

Exam 

Locally- 

Developed 

Exam 

A^- 

\^c'  ,eo<"'1'  p.YC'0  \ 

A.  \ 

Capstone 

Experience 

/I 

Portfolio 

Direct -Grad 
Performance 

V  c0^e 

\  *WZC  ..Wv<\ed 

Archival  Data 

_ A  A 

Stakeholder 

Feedback 

o£e 

Supervisor 
Interview  / 
Survey 

Indirect 

Journals  / 
Reflections 

Course 

Feedback 

Cadet  Survey 

Focus 

Groups 

Graduate 
Interview  / 
Survey 
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Chapter  6  -  Making  Sense  of  Assessment  Results 

At  this  point  in  the  assessment  process,  your  program  has  presumably  collected  data  related  to  how 
well  cadets  are  achieving  your  programmatic  cadet  learning  outcomes.  However,  your  work  is  still  far 
from  done.  In  fact,  it  is  now  time  to  tackle  some  of  the  most  important,  most  rewarding,  and  -  in  some 
cases  -  most  overlooked  aspects  of  the  assessment  process:  making  sense  of  the  data.  This  chapter  is 
written  to  provide  some  practical  guidance  regarding  this  important  sense-making  activity.  The  first  part 
of  the  chapter  focuses  on  compiling  the  various  sources  of  data  that  are  available  to  your  program.  The 
second  part  of  the  chapter  focuses  on  reporting  the  data  in  a  form  that  is  easily  interpretabie.  Finally, 
the  third  part  of  the  chapter  provides  some  tips  on  how  to  make  sense  of  the  data  as  they  specifically 
pertain  to  cadets'  achievement  of  your  programmatic  learning  outcomes. 

Compiling  the  Available  Data 

If  your  program  is  like  most,  you  do  not  suffer  from  a  lack  of  information  about  cadets'  learning.  In  fact, 
a  quick  review  of  the  Summary  of  Assessment  Methods,  by  Academic  Major  {see  Table  7,  page  49} 
indicates  that  the  vast  majority  of  programs  have  access  to  many,  many  different  sources  of  assessment 
data.  One  key  to  success,  then,  is  to  compile  all  of  the  data  and  report  them  in  a  format  that  you  and 
your  departmental  colleagues  can  digest.  How  can  this  be  done? 

A  good  first  step  is  to  get  all  of  the  relevant  assessment  data  pertaining  to  each  of  your  outcomes 
together  in  one  place.  This  is  a  relatively  simple  step,  but  it  is  very  important,  particularly  given  that 
assessment  data  relevant  to  any  given  outcome  may  come  from  a  variety  of  different  sources.  Some 
may  come  from  course-embedded  tests  and  assignments  distributed  across  your  academic  classes. 
Others  may  come  from  institutional-level  tests  or  surveys.  Still  more  may  come  from  interviews  or  focus 
groups  conducted  with  your  graduates.  In  order  to  make  sense  of  these  disparate  pieces  of  data,  it  will 
first  be  necessary  to  gather  the  relevant  data  into  one  place. 

As  indicated  above,  one  common  source  of  assessment  data  is  the  collection  of  tests  and  assignments 
embedded  within  academic  courses.  Those  data  can  be  incredibly  valuable,  as  they  most  accurately 
reflect  the  work  that  cadets  are  doing  in  your  program's  courses.  In  addition,  data  that  come  from 
embedded  assessments  are  relatively  easy  to  collect.  Cadets  are  already  doing  the  work;  faculty 
members  are  already  evaluating  it;  and  your  curriculum  map  will  help  you  know  where  to  look  for  the 
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information.  However,  given  that  the  data  will  still  be  coming  from  multiple  course  directors  and 
instructors,  a  primary  challenge  may  be  compiling  the  data  at  the  conclusion  of  each  semester, 

Currently,  our  academic  departments  use  a  variety  of  processes  to  compile  course-embedded 
assessment  data.  Here  are  some  notable  examples  of  this  process  in  use  around  DF: 

•  In  the  Department  of  Behavioral  Sciences  and  Leadership  (DFBL),  all  course  directors  are 
required  to  write  an  end-of-course  assessment  report-  To  ensure  consistency  across  courses, 
these  reports  follow  a  standard  template  in  which  course  directors  must  summarize  the 
assessment  data  collected  in  their  course.  Those  reports  are  submitted  and  reviewed  by  the 
"Discipline  Lead"  overseeing  each  course. 

•  In  the  Department  of  Chemistry,  course  directors  are  expected  to  include  assessment  results 
within  larger  Course  Diaries.  Each  diary  includes  a  variety  of  information  about  the  course,  to 
include  a  course  description,  a  list  of  textbooks,  and  the  number  of  cadets  enrolled  in  the 
course.  In  addition,  diaries  also  include  the  programmatic  outcomes  assessed  in  the  course,  the 
results  of  those  assessments,  and  a  narrative  from  the  Course  Director  identifying 
recommendations  for  course  improvement. 

•  The  Department  of  Management  (DFM)  uses  a  LOSAn  system  to  compile  their  assessment 
results.  At  the  beginning  of  each  semester,  course  directors  are  required  to  articulate  which  of 
the  management  major's  learning  objectives  they  are  covering  and/or  assessing  in  their  courses. 
At  the  end  of  the  semester,  course  directors  present  the  results  of  their  assessments  and  offer 
suggestions  for  improvement  at  a  department-wide  meeting. 

Realistically,  the  best  way  to  collect  assessment  data  from  academic  courses  varies  from  program  to 
program.  You  are  encouraged  to  find  a  method  that  works  best  (and  is  most  sustainable)  within  your 
particular  departmental  culture.  However,  regardless  of  the  details  of  how  the  assessment  data  are 
reported,  it  is  likely  that  the  information  collected  will  be  largely  the  same  across  all  programs.  Typical 
information  you  may  wish  your  course  directors  to  provide  are  summarized  in  the  generic  "Course 
Reporting  Template"  shown  on  page  52.  Please  feel  free  to  adopt  this  form  for  your  own  program's  use 
or  modify  it  in  a  way  that  makes  sense  for  your  particular  situation. 


LOSA  =  Learning  Objectives  Statement  of  Assessment 
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SAMPLE  COURSE  REPORTING  TEMPLATE 

Course: _ 

Course  Director: _ 

Semester: _ 

Programmatic  Cadet  Learning  Outcome  #1: _ 

•  How  was  this  cadet  learning  outcome  developed  in  this  course?  (For  example,  what  readings, 
lessons,  activities,  etc.  were  built  into  the  course  to  help  cadets  achieve  this  outcome?) 

•  How  was  this  cadet  learning  outcome  assessed  in  this  course?  Please  provide  examples  of  exam 
questions,  assignment  prompts,  etc. 

•  What  were  the  results?  Please  summarize  the  assessment  data  collected  in  this  course. 

•  How  do  the  assessment  results  compare  to  your  expectations?  Are  cadets  achieving  this 
outcome  as  you  expected,  or  are  they  falling  short  in  one  or  more  areas? 

•  If  appropriate,  what  would  you  recommend  be  done  to  improve  cadet  accomplishment  of  this 
outcome? 

Programmatic  Cadet  Learning  Outcome  #2: _ 

•  How  was  this  cadet  learning  outcome  developed  in  this  course?  (For  example,  what  readings, 
lessons,  activities,  etc.  were  built  into  the  course  to  help  cadets  achieve  this  outcome?) 

•  How  was  this  cadet  learning  outcome  assessed  in  this  course?  Please  provide  examples  of  exam 
questions,  assignment  prompts,  etc. 

•  What  were  the  results?  Please  summarize  the  assessment  data  collected  in  this  course. 

•  How  do  the  assessment  results  compare  to  your  expectations?  Are  cadets  achieving  this 
outcome  as  you  expected,  or  are  they  falling  short  in  one  or  more  areas? 

•  If  appropriate,  what  would  you  recommend  be  done  to  improve  cadet  accomplishment  of  this 
outcome? 


Etc. 
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Of  course,  some  of  the  assessment  data  pertaining  to  cadet  accomplishment  of  your  programmatic 
outcomes  will  not  come  from  course-embedded  tests  and  assignments.  Assessment  data  can  also  come 
from  a  variety  of  other  sources.  For  instance,  several  of  the  majors  in  our  Engineering  Division  rely 
heavily  on  the  Fundamentals  of  Engineering  exam  as  a  source  of  assessment  data.  Other  programs  rely 
on  institutional  surveys  (e.g.,  the  National  Survey  of  Student  Engagement)  or  focus  groups  to  provide 
information  about  cadets'  accomplishment  of  their  learning  outcomes.  Obviously,  your  program  will 
want  to  make  good  use  of  these  data  sources  as  well.  This  can  be  done  by  adapting  the  questions  on  the 
Course  Reporting  Template  for  use  outside  of  a  normal  course.  An  example  is  shown  in  the  Sample 
Assessment  Data  Reporting  Template  below: 


SAMPLE  ASSESSMENT  DATA  REPORTING  TEMPLATE 
Name  of  POC: _ _ 

Semester: _ _ _ 

Programmatic  Cadet  Learning  Outcome  #1: _ _ _ 

•  How  was  this  cadet  learning  outcome  assessed?  What  assessment  method(s)  was  used? 

•  What  were  the  results?  Please  summarize  the  assessment  data  collected. 

•  How  do  the  assessment  results  compare  to  our  expectations?  Are  cadets  achieving  this 
outcome  as  we  expect,  or  are  they  falling  short  in  one  or  more  areas? 

•  If  appropriate,  what  would  you  recommend  be  done  to  improve  cadet  accomplishment  of 
this  outcome? 
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Summarizing  the  Results 

While  compiling  assessment  data  from  various  sources  is  valua  ble,  the  mere  compilation  of  that 
information  is  not  sufficient.  You  will  also  want  to  ensure  that  those  data  are  summarized  in  a  form  that 
will  be  useful  to  you12.  This  section  includes  three  broad  tips  to  get  you  pointed  in  the  right  direction. 

Tip  Ul;  Consider  using  tables  or  graphs  to  summarize  your  results.  Piles  and  piles  of  assessment  data  can 
be  difficult  to  digest.  Therefore,  it  is  important  that  someone  summarize  the  data  from  each  assessment 
in  a  format  that  accurately  conveys  the  key  results.  This  can  be  done  in  a  variety  of  ways,  many  of  which 
you  probably  learned  in  an  introductory  statistics  course  at  some  point  in  your  academic  career.  These 
include  (but  are  certainly  not  limited  to): 

•  Providing  frequency  counts,  such  as  the  number  of  cadets  scoring  at  different  levels  on  a 
particular  test  or  assignment.  These  counts  can  be  presented  in  the  form  of  either  a  table  or  a 
simple  graph.  For  instance,  the  table  in  the  left  panel  below  shows  the  hypothetical  distribution 
of  scores  that  cadets  may  have  on  a  specific  portion  of  a  classroom  exam.  The  panel  on  the  right 
represents  the  same  data  in  graphical  form. 


Frequency  of  Scores  on  a  Specific  Portion  of  a  Classroom  Exam 


Below  60  60-69  70-79  80-89  90-100 

Score 


12  One  of  the  biggest  challenges  that  many  beginners  to  assessment  might  face  is  having  a  huge  collection  of 
assessment  data  and  not  knowing  exactly  what  to  do  with  it.  If  you  find  yourself  in  this  situation,  rest  assured  that 
you  are  not  alone.  This  section  is  written  to  provide  some  tips  on  how  you  can  proceed.  Your  colleagues  within  DF 
can  also  provide  assistance  in  helping  you  move  forward. 
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*  Reporting  average  scores,  a  strategy  that  is  particularly  useful  if  a  single  number  does  a 

reasonably  good  job  of  summarizing  your  data13.  Reporting  averages  can  also  be  useful  if  you 
wish  to  compare  the  performance  of  different  subgroups  of  cadets.  For  instance,  the  graph 
below  compares  the  average  scores  of  cadets  in  different  class  years  on  a  hypothetical  attitude 
scale. 


Attitude  Score,  By  Class 

100  j- - - - - 

90 - — — - — ™ 


S  70 


4th  Class  3rd  Class  2nd  Class  1st  Class 


»  Reporting  the  percentage  of  cadets  who  score  above  a  certain  threshold.  Imagine,  for  instance, 
that  your  program  administers  a  standardized  test  to  first-class  cadets  at  the  end  of  each  Spring 
semester.  One  strategy  could  be  to  report  the  percentage  of  cadets  who  pass  the  test  (where 
the  criteria  for  passing  would  need  to  be  dearly  defined)  each  year,  as  shown  in  the  graph 
below. 


13  If  your  data  approximate  a  normal  distribution,  for  example,  an  arithmetic  mean  can  do  a  good  job  of 
summarizing  the  central  tendency  of  your  data  with  a  single  number. 
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Class  of  Class  of  Class  of  Class  of  Class  of  Class  of 
2009  2010  2011  2012  2013  2014 


•  Reporting  summaries  of  qualitative  data.  Some  of  the  data  you  collect  may  not  be  quantitative 
in  nature.  For  example,  your  program  may  gain  a  great  deal  of  information  from  examining 
cadets'  written  feedback,  their  responses  to  interview  questions,  or  their  comments  in  focus 
groups.  None  of  that  information  will  necessarily  come  in  the  form  of  numbers,  yet  you  will  still 
want  to  find  a  way  to  summarize  it  in  an  easy-to-digest  form*  One  way  to  do  that  is  to  perform  a 
thematic  analysis,  where  you  sort  cadets'  responses  into  categories  based  on  the  content.  For 
instance,  if  you  were  to  ask  cadets  what  experiences  have  been  most  beneficial  in  their 
development  of  a  particular  outcome,  you  may  find  that  their  responses  fall  into  categories  like 
the  following: 

o  Category  1:  Core  Courses 
o  Category  2:  Majors'  Courses 
o  Category  3:  Your  Program's  Capstone  Experience 
o  Category  4:  Research  Experiences 

o  Category  5:  Extracurricular  Experiences  (e.g,,  clubs,  teams,  etc,) 

After  sorting  cadets'  comments  into  these  different  categories,  you  may  also  find  it  helpful  to 
tally  the  number  of  comments  that  fat!  within  each  one.  For  instance,  if  most  of  your  cadets 
indicated  that  research  experiences  have  been  beneficial,  but  relatively  few  indicate  that 
majors'  courses  have  been  beneficial,  you  may  wish  to  examine  what  happens  in  those  research 
experiences  and  determine  if  it  is  possible  to  export  some  of  those  best  practices  to  those 
majors'  courses  where  those  practices  don't  already  exist. 
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Tip  #2:  Hove  something  to  compare  your  results  to.  By  itself,  a  given  assessment  result  may  not  be 
terribly  informative.  For  instance,  imagine  that  your  cadets  have  an  average  score  of  75%  on  a  particular 
test,  or  imagine  that  60%  of  your  cadets  scored  "proficient"  or  above  on  a  particular  assignment.  Are 
these  results  something  that  you  can  celebrate?  Are  they  merely  satisfactory?  Or  are  they  cause  for 
concern?  Unfortunately,  you  don't  know  unless  you  also  have  some  standard  you  can  use  for 
comparison,  A  key  part  of  making  sense  of  assessment  data  is  to  make  it  clear  what  that  standard  is. 

Here  is  a  sampling  of  some  standards  you  might  use  in  making  sense  of  the  data  in  your  program. 

•  Comparisons  between  different  groups  of  cadets.  Imagine  that  you  and  your  colleagues  are 
trying  to  determine  the  extent  to  which  cadets  retain  the  knowledge  from  a  particular  course. 
One  way  to  approach  this  question  would  be  to  give  first-class  cadets  a  standardized  test  (such 
as  ETS'  Major  Field  Test)  that  includes  questions  related  to  that  course  and  then  examine  cadets' 
performance  on  those  specific  questions.  In  addition,  it  would  be  wise  to  compare  the  scores  on 
those  questions  for  those  cadets  who  have  taken  the  course  in  question  with  those  cadets  who 
have  not*  If  the  course  had  a  lasting  impact  on  cadets'  knowledge,  you  would  expect  that  cadets 
who  have  taken  the  course  would  score  significantly  higher  than  those  who  have  not  done  so. 

On  the  other  hand,  if  cadets  have  forgotten  much  of  what  they  had  learned  in  the  course,  then 
you  might  find  that  cadets  who  have  taken  the  course  perform  no  better  than  those  who 
haven't. 

•  Comparisons  over  time.  If  your  program  is  in  the  habit  of  collecting  data  using  the  same 
assessment  instrument  from  semester-to-semester  or  year-to-year,  it  is  possible  to  compare 
cadets'  current  scores  on  that  instrument  to  cadets'  scores  in  previous  administrations  of  the 
instrument.  For  instance,  imagine  that  you  are  trying  to  cultivate  scientific  thinking  among 
cadets  in  your  program,  and  you  are  in  the  habit  of  administering  a  measure  of  scientific 
reasoning  to  first-class  cadets  each  year  before  they  graduate.  You  could  easily  track  cadets' 
performance  from  year  to  year,  taking  note  of  what  changes  in  your  program  seem  to  lead  to 
upward  or  downward  shifts*  As  you  learn  more  and  more  about  how  to  develop  this  type  of 
thinking  in  your  cadets,  one  would  hope  that  scores  on  this  measure  would  gradually  rise  over 
time, 

•  Comparisons  between  cadets  and  students  at  peer  institutions.  While  many  of  the  assessment 
instruments  in  use  at  USAFA  are  generated  internally,  there  are  some  that  are  commercially 
available  for  use  at  other  schools.  In  those  cases,  you  may  find  if  beneficial  to  compare  the 
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scores  of  cadets  in  your  program  to  the  scores  of  students  at  other  institutions.  An  obvious 
example  of  this  involves  our  cadets'  responses  on  the  National  Survey  of  Student  Engagement 
(NSSE),  a  survey  that  we  administer  at  USAFA  every  three  years14.  The  NSSE  is  given  at  hundreds 
of  colleges  and  universities  nationwide  each  year,  to  include  at  dozens  of  other  highly  selective 
undergraduate  institutions  and  each  of  the  other  military  service  academies.  As  a  result,  in 
addition  to  examining  our  institutional  results  {broken  out  by  academic  major),  it  is  also 
beneficial  to  compare  our  results  with  the  results  from  institutions  that  are  similar  to  us.  Of 
course,  we  would  like  it  if  our  cadets  gave  responses  on  the  NSSE  that  were  as  favorable  as,  or 
more  favorable  than,  responses  given  by  students  at  peer  institutions.  However,  in  cases  where 
they  don't,  those  data  could  provide  a  clue  as  to  where  we  may  need  to  improve  our  processes. 

«  Comparisons  with  what  we  want  or  expect.  Even  if  you've  created  an  assessment  instrument  in- 
house  (meaning  that  it  doesn't  have  norms  from  other  institutions)  and  you're  using  it  for  the 
first  time  (meaning  that  you  lack  historical  norms  as  well),  it  is  still  beneficial  to  have  a  standard 
against  which  you  can  compare  your  results.  In  these  cases,  that  standard  is  likely  to  come  from 
your  professional  judgment  and  the  judgment  of  your  program's  faculty  members.  For  instance, 
given  the  emphasis  your  department  has  placed  on  improving  cadet  writing,  you  might  expect 
that  80%  of  your  cadets  will  achieve  at  a  proficient  level  on  a  particular  writing  assignment.  If 
cadets'  performance  meets  that  standard,  then  you  can  celebrate.  However,  if  cadets  fall  short 
of  that  expectation,  it  might  be  a  cue  for  your  program  to  take  another  look  at  what  you're 
doing  to  improve  cadet  writing  and  whether  there  are  ways  to  do  it  better. 

Tip  #3:  Stay  focused  on  your  cadet  learning  outcomes.  This  last  tip  may  seem  obvious,  but  it  is  important 
enough  that  it  is  still  worth  emphasizing.  The  key  is  to  think  about  why  you  and  your  colleagues  are 
collecting  assessment  data  in  the  first  place.  You  shouldn't  be  collecting  data  just  for  the  sake  of 
collecting  assessment  data.  Instead,  the  purpose  of  that  data  collection  is  to  inform  you  and  your  peers 
about  the  degree  to  which  cadets  are  progressing  toward  successful  accomplishment  of  your  stated 
outcomes.  Therefore,  the  data  you  collect  will  be  useful  only  to  the  extent  that  they  provide  meaningful 
information  about  your  cadets'  outcome  achievement.  Furthermore,  any  time  you  spend  examining 
those  data  ought  to  focus  on  what  the  data  tell  you  about  cadets'  achievement  of  the  outcomes. 


14  The  most  recent  administration  of  the  NSSE  was  in  May  of  2014. 
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Let's  consider  a  specific  example  that  may  be  salient  in  your  program.  Many  departments  closely 
monitor  the  grades  earned  by  cadets  in  each  of  their  classes,  paying  particular  attention  to  historical 
trends  and  whether  current  grades  are  consistent  with  those  historical  norms.  This  practice  may  be 
quite  helpful  administratively,  but  it  is  not  likely  to  provide  very  much  useful  information  about  how  well 
cadets  are  progressing  toward  your  programmatic  learning  outcomes.  The  reason  is  threefold: 

•  Many  departments  use  grades  as  an  indicator  of  cadets'  relative  performance  (i.e.,  compared  to 
their  peers)  in  the  course.  The  students  who  perform  best  receive  A's,  the  students  who 
perform  at  a  good  level  receive  B's,  students  who  perform  more  poorly  receive  C's,  etc. 
However,  grades  may  not  say  very  much  about  cadets'  absolute  performance  (i.e.,  against  some 
consistent  standard).  The  exception  is  obviously  in  those  cases  where  departments  consistently 
require  particular  graded  events  from  semester  to  semester  and  have  processes  in  place  to 
evaluate  cadets'  performance  on  those  graded  events  consistently  as  well.  In  those  cases,  one 
could  argue  that  a  "B"  earned  during  one  semester  means  a  level  of  absolute  performance  that 
is  comparable  to  a  "B"  earned  during  a  different  semester. 

•  Many  academic  courses  are  aligned  with  multiple  cadet  learning  outcomes,  and  grades  in  those 
courses  reflect  cadets'  accomplishment  of  those  outcomes  simultaneously.  For  instance,  if  a 
particular  course  is  designed  to  develop  both  cadets'  research  skills  and  their  writing  skills,  then 
cadets'  grades  will  likely  reflect  how  well  they  conduct  research  and  how  well  they  write.  In  such 
cases,  a  grade  of  "B"  could  mean  multiple  things.  Cadets  could  be  outstanding  researchers  but 
only  mediocre  writers;  they  could  be  mediocre  researchers  but  outstanding  writers;  or  they 
could  be  good  (but  not  outstanding)  at  both  skills.  Without  disaggregating  the  course  grades 
into  these  two  components,  it  would  be  impossible  to  know  how  cadets  are  doing  in  each 
specific  area. 

•  Often,  grades  are  influenced  by  things  that  having  nothing  to  do  with  outcome  achievement  at 
all.  As  one  example,  many  departments  have  a  policy  whereby  cadets  receive  a  reduced  grade 
on  assignments  that  are  submitted  late.  Therefore,  it  is  possible  for  a  cadet  to  be  an  outstanding 
writer  but  receive  a  poor  grade  on  a  paper  because  he  failed  to  turn  his  assignments  in  on  time. 
As  a  result,  merely  looking  at  the  cadets'  grades  wouldn't  provide  you  with  accurate  information 
about  their  writing  skills  specifically. 

Rather  than  examining  course  grades  as  a  source  of  information  about  outcome  achievement,  a 
different  approach  appears  necessary.  One  strategy  would  be  to  examine  more  specific  aspects  of 
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cadets'  performance.  For  instance,  if  writing  is  indeed  one  of  your  programmatic  outcomes,  perhaps 
your  instructors  could  evaluate  cadets'  papers  (or  at  least  a  sample  of  cadets'  papers)  specifically  for 
how  well  they  are  written.  Cadets  would  still  receive  overall  grades  that  may  reflect  other  aspects  of 
their  performance  (e.g.,  quality  of  research,  timeliness  of  submission,  etc.},  but  the  data  that  would  be 
passed  forward  for  assessment  purposes  would  focus  more  specifically  on  cadets'  writing  skills.  Those 
are  the  data  that  would  be  most  beneficial  when  considering  the  level  of  cadets'  outcome  achievement. 

Reviewing  Your  Results  in  Light  of  the  Outcomes 

The  final  step  in  making  sense  of  the  data  you've  collected  is  to  have  someone  (or,  preferably,  a  group  of 
people)  review  the  data  pertaining  to  each  of  the  programmatic  learning  outcomes.  That  may  mean 
asking  the  entire  faculty  to  review  the  data,  perhaps  at  a  faculty  offsite.  Alternatively,  it  could  mean 
engaging  the  members  of  the  department's  Curriculum  Committee.  Or  it  could  mean  creating  several 
"outcome  committees"  for  your  academic  program,  with  each  committee  dedicated  to  reviewing  and 
evaluating  the  assessment  data  pertaining  to  one  particular  learning  outcome.  As  with  many  aspects  of 
the  assessment  process,  there  is  no  single  right  structure  for  reviewing  assessment  data.  The  key  is 
simply  to  have  something  that  fits  within  the  culture  of  your  department  or  program. 

Even  though  the  structure  that  reviews  assessment  data  may  vary  from  program  to  program,  it  is  likely 
that  the  task  of  this  group  will  largely  be  the  same.  Specifically,  they  are  likely  to  ask  the  questions 
included  in  the  "Sample  Outcome  Reporting  Template,"  included  on  page  61.  For  each  outcome,  they 
will  want  to  review  how  that  outcome  is  developed  within  your  academic  program;  provide  a  summary 
of  how  that  outcome  was  assessed;  offer  a  summary  of  the  major  assessment  results;  interpret  what  the 
results  mean  within  the  context  of  your  expectations;  and  -  if  applicable  -  make  recommendations  for 
changes  in  departmental  processes  that  could  improve  cadets'  achievement  in  the  future. 

In  the  name  of  documenting  your  assessment  processes,  I  would  urge  your  program  to  create  written 
records  of  your  outcome  reviews.  Those  reviews  will  help  you  keep  track  of  assessment  results  from 
semester  to  semester,  and  they  will  also  be  very  helpful  as  USAFA  builds  their  case  for  institutional 
accreditation  during  the  next  review  of  the  Filgher  Learning  Commission. 
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SAMPLE  OUTCOME  REPORTING  TEMPLATE 


Programmatic  Learning  Outcome: 


Curriculum  Map:  What  courses  are  aligned  with  this  outcome  in  our  curriculum  map?  What  happens  in 
those  courses  to  help  cadets  develop  this  learning  outcome? 


Assessment  Data:  What  data  do  we  have  that  provide  information  about  how  well  cadets  are  achieving 
this  outcome?  Where  /  When  /  How  was  each  set  of  data  collected? 


Results:  Summarize  the  assessment  data  pertaining  to  this  outcome, 


Interpretation:  What  do  the  results  mean?  How  do  they  compare  with  our  expectations? 


Suggestions  /  Implications:  Based  upon  our  findings,  what  could  /  should  be  done  to  improve  cadets' 
accomplishment  of  this  outcome? 
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Of  course,  while  accreditation  is  important,  it  is  not  the  most  important  reason  for  engaging  in  a 
systematic  review  of  your  assessment  data,  A  far  more  important  reason  is  that  the  assessment  process 
allows  you  and  your  department  to  make  data-informed  improvements  within  your  department  or 
program.  To  maximize  the  chances  of  having  a  positive  impact,  here  are  some  general  tips  I  encourage 
you  to  use  when  making  sense  of  assessment  data  and/or  generating  your  outcome  reports15. 

•  Get  lots  of  people  involved  in  making  sense  of  the  data.  If  making  sense  of  the  data  is  the  job  of 
one  person  or  a  very  small  group,  it  is  likely  that  many  of  your  faculty  will  feel  disconnected 
from  the  process  and  less-than-fully  invested  in  the  results,  if  a  larger  number  of  people  have  a 
chance  to  review  the  data,  they  will  be  more  apt  to  own  the  results  and  also  more  apt  to  act 
upon  them. 

•  Be  transparent  with  the  data.  Faculty  members  are  unlikely  to  trust  conclusions  if  they  don't 
know  where  they  come  from.  Be  clear  about  where  the  data  came  from  and  how  you  arrived  at 
your  conclusions.  This  will  make  any  calls  for  change  much  easier  for  faculty  to  comprehend. 

®  Don't  be  afraid  to  use  informed  judgment.  Assessment  results  are  far  more  than  numbers  on  a 
page.  They  are  part  of  a  narrative  about  your  cadets'  learning,  and  making  sense  of  them  - 
particularly  when  they  come  from  multiple  sources  -  requires  thoughtful  interpretation  on  the 
part  of  your  faculty  members.  Thankfully,  you  and  your  faculty  colleagues  are  better  equipped 
than  anyone  else  to  interpret  the  data  within  the  context  of  your  program.  They  should  feel 
empowered  to  offer  informed  commentary  on  the  assessment  results. 

•  Keep  it  simple.  Sometimes,  assessment  results  really  are  complex,  and  they  take  a  great  deal  of 
time,  energy,  and  space  to  explain.  More  often  than  not,  however,  the  most  important 
conclusions  are  those  that  are  relatively  straightforward.  This  means  that  outcome  reports  may 
actually  be  fairly  brief.  That  will  make  them  markedly  easier  to  write,  and  it  will  also  make  it 
more  likely  that  people  in  your  department  wifi  read  and  act  on  them. 

•  Keep  your  eyes  on  the  prize.  Throughout  the  assessment  process,  it  is  important  to  remember 
why  you  are  doing  this  in  the  first  place.  Our  primary  driver  is  the  improvement  of  cadet 
learning.  Your  interpretations  of  assessment  data,  like  the  data  themselves,  wifi  be  useful  to  the 
extent  that  it  leads  to  improvements  in  what  your  cadets  know  and  are  able  to  do.  This  is  a  point 
that  I  will  expand  on  in  the  final  chapter  of  this  Handbook. 

5  These  tips  are  based  on  information  in  an  Assessment  Workbook  written  by  the  Office  of  Institutional 
Effectiveness  at  Bail  State  University.  That  handbook  is  available  online  at: 
http://cms.bsu.edu/about/administrativeoffices/effectiveness/assessmentresources/workbook. 
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This  chapter  ends  with  the  fourth  dimension  ("Making  Sense  of  Assessment  Results")  of  the  Academic 
Program  Seif-Assessment.  That  dimension  is  shown  in  Table  8  on  page  64. 
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Table  8 

Departmental  Self-Assessment 
Making  Sense  of  Assessment  Results 


This  table  includes  one  row  of  the  broader  Academic  Program  Self-Assessment  rubric  that  has  been  in  place  within  DF.  (The  rubric  is  printed  in 
its  entirety  in  Appendix  A.)  In  conjunction  with  your  colleagues,  identify  which  description  (i.e.,  Level  0, 1,  2,  or  3)  best  matches  the  current  state 
of  cadet  learning  outcomes  within  your  academic  major  or  minor. 


Elements 

Question(s)  To 
Think  About 

Level  0 

Level  1 

Level  2 

Level  3 

Making 

Sense  of  the 
Results 

Has  my 
department 
established 
processes  for 
compiling  and 
making  sense  of 
assessment  results 
for  each  learning 
outcome? 

No  efforts 
are  being 
made  to 
compile  or 
make  sense 

of 

assessment 

results. 

Assessment  results  are 
compiled  from  relevant 
sources  (e.g.,  in  the  form 
of  course  diaries  or 
course  director 
memoranda),  but  no  one 
reviews  the  compilation 
as  a  way  to  better 
understand  how  well 
cadets  are  achieving  the 
learning  outcomes. 

Assessment  results  are 
compiled  from  relevant 
sources,  and  the 
department  has  begun  to 
create  a  process  whereby 
faculty  review  those  results 
for  a  better  understanding 
of  how  well  cadets  are 
achieving  at  least  one 
learning  outcome. 

Assessment  results  are  compiled 
from  relevant  sources,  and  the 
department  has  established  a 
process  whereby  faculty  review 
those  results  for  a  better 
understanding  of  how  well 
cadets  are  achieving  each  of  the 
learning  outcomes. 
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Chapter  7  -  Using  Assessment  for  Improvement 

Just  over  ten  years  ago,  George  Kuh,  then  the  Director  of  Indiana  University's  Center  for  Postsecondary 
Research,  led  an  ambitious  effort  (called  "Documenting  Effective  Educational  Practices"  or  "DEEP")  to 
identify  and  characterize  some  of  America's  highest-performing  colleges  and  universities,  Kuh's  research 
team  examined  both  graduation  rates  and  institutional  scores  on  the  National  Survey  of  Student 
Engagement,  ultimately  identifying  20  institutions  that  scored  higher  than  one  would  expect  based  on 
the  demographic  characteristics  of  their  student  bodies.  His  team  then  visited  each  of  the  20  high- 
performing  campuses,  interviewed  faculty  and  students,  and  did  their  best  to  describe  the 
characteristics  that  made  these  institutions  so  effective.  In  brief,  his  team  discovered  that  the  20 
colleges  and  universities  in  the  study  had  six  significant  things  in  common16. 

1*  All  of  the  DEEP  schools  have  a  clearly-articulated  mission  that  guides  how  /  why  things  are  done 
on  their  campuses. 

2.  All  of  the  DEEP  schools  have  an  "unshakeable  focus  on  student  learning."  (Kuh,  Kinzie,  Schuh,  & 
Whitt,  2005,  p.  65) 

3.  All  of  the  DEEP  schools  use  their  distinctive  campus  environments  to  improve  student  learning. 

4.  All  of  the  DEEP  schools  provide  students  with  "clear  pathways"  to  success  (Kuh  et  aL,  2005,  p. 
109}. 

5.  All  of  the  DEEP  schools  have  a  sense  of  "positive  restlessness"  that  drives  them  to  continuously 
improve  (Kuh  et  aL,  2005,  p.  146). 

6.  All  of  the  DEEP  schools  involve  faculty  and  staff  from  across  the  institution  in  promoting  student 
success. 

All  of  the  characteristics  described  above  are  worthy  of  emulating,  and  -  thankfully  -  many  of  them  are 
already  be  in  place  here  at  USAFA.  For  the  purposes  of  this  chapter  of  the  Assessment  Handbook , 
however,  I  would  call  your  attention  to  Characteristic  #5  about  "positive  restlessness"  and  continuous 
improvement.  Kuh  et  aL  (2005)  discovered  that  faculty  and  staff  at  high-performing  institutions  are 
confident  in  who  they  are  and  what  they  are  trying  to  accomplish,  but  they  are  constantly  searching  for 
ways  to  get  better.  Said  one  faculty  member  interviewed  for  the  study,  "We  talk  about  what  needs  to  be 
fixed  all  the  time.  This  is  very  much  a  part  of  our  culture"  (p.  146).  They  are  restless,  not  in  a  cynical  or 
complaining  way,  but  in  a  positive  way  that  catalyzes  them  to  continuously  improve* 


lDThe  complete  summary  of  the  DEEP  project  is  available  in  Kuh,  Kinzie,  Schuh,  &  Whitt  (2005), 
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It  is  in  the  spirit  of  positive  restlessness  that  I  have  written  this  Assessment  Handbook,  and  it  is  that 
same  spirit  that  i  would  encourage  you  to  adopt  as  you  proceed  with  the  assessment  work  going  on  in 
your  academic  program.  There  is  no  doubt  that  faculty  and  staff  in  your  program  are  doing  terrific 
things,  both  in  and  out  of  the  classroom.  There  is  also  no  doubt  that  the  cadets  are  benefiting 
substantially  from  their  efforts.  However,  it  is  almost  certainly  the  case  that  there  are  things  that  you 
and  your  colleagues  could  do  to  help  cadets  learn  even  better  than  they  are  currently.  As  such,  I 
encourage  you  to  use  the  assessment  processes  described  in  this  Handbook  as  tools,  both  to  identify 
those  aspects  of  cadet  learning  that  are  most  in  need  of  improvement  and  to  help  your  department 
figure  out  how  to  put  needed  improvements  into  place. 

A  Quick  Review  of  Where  We've  Been 

In  Part  II  of  this  Handbook,  you've  worked  through  four  very  important  steps  of  the  assessment  process. 
Specifically,  you've  learned  how  to: 

•  Create  cadet  learning  outcomes  for  your  academic  program  (Chapter  3); 

•  Build  a  map  between  your  curriculum  and  those  stated  learning  outcomes  (Chapter  4); 

•  Select  assessment  methods  that  will  provide  you  with  credible  information  about  how  well 
cadets  are  achieving  those  outcomes  (Chapter  S};  and 

•  institute  processes  for  making  sense  of  your  assessment  results  (Chapter  6). 

Once  you've  completed  these  first  four  steps,  it  is  time  to  use  what  you've  learned  from  your 
assessment  results  to  improve  the  quality  of  cadet  learning  within  your  academic  program.  That  is,  it  is 
time  to  implement  the  changes  that  you  and  your  departmental  colleagues  recommended  in  your 
outcome  reporting  process  (see  p.  61}.  As  you  move  forward,  I  would  urge  you  to  closely  monitor  those 
changes  to  see  if  they  worked.  If  they  did,  it  is  once  again  time  to  celebrate.  If  they  didn't,  it  will  be  a 
due  that  still  more  changes  may  be  necessary. 

A  Case  Study 

To  illustrate  the  entire  assessment  process,  consider  a  hypothetical  academic  department  that  is  new  to 
the  process  of  conducting  assessment.  Based  on  the  guidance  provided  in  this  Handbook,  faculty  and 
staff  in  that  department  may  proceed  as  follows: 
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•  First,  faculty  and  staff  may  get  together  to  decide  on  suitable  learning  outcomes  for  their  major. 
Based  on  discussions  within  the  department,  consultation  with  Air  Force  constituencies,  and 
input  from  a  professional  organization  in  their  discipline,  let's  assume  that  one  of  the  outcomes 
they  settle  on  can  be  written  as  follows: 

"As  a  result  of  being  in  this  program,  cadets  will  write  effective  research  papers." 

•  Second,  faculty  and  staff  may  build  a  curriculum  map,  such  that  they  identify  a  number  of 
required  courses  within  their  major  that  will  provide  cadets  with  substantia!  instruction  in  how 
to  write  effective  research  papers.  For  the  sake  of  this  example,  let's  assume  that  there  are 
three  such  courses,  one  each  at  the  200,  300,  and  400  levels.  This  means  that  cadets  in  the 
program  will  receive  substantial  instruction  (as  well,  as  substantial  practice  and  feedback,  I 
presume)  on  writing  effective  research  papers  at  least  once  each  year. 

•  Third,  department  members  will  need  to  identify  assessment  methods  that  can  inform  them 
about  how  well  cadets  are  achieving  the  outcome  of  writing  effective  research  papers. 

o  In  this  example,  one  of  the  most  obvious  sources  of  assessment  data  may  be  the  papers 
that  cadets  write  in  the  courses  shown  in  the  curriculum  map.  Therefore,  the 
department  may  ask  instructors  in  those  courses  to  grade  the  papers  using  a  rubric  that 
is  applied  across  all  departmental  courses.  Course  directors  could  then  summarize  the 
results  using  the  Course  Reporting  Template  on  p.  52.1' 

o  The  department  may  decide  to  supplement  the  direct  measures  of  cadet  learning  with 
indirect  measures  as  well.  So,  they  may  ask  cadets  to  comment  in  end-of-course 
feedback  and  in  focus  groups  about  what  aspects  of  their  courses  are  most  helpful  in 
learning  to  write  effective  research  papers  and  what  aspects  are  most  in  need  of 
improvement.  A  POC  may  be  asked  to  summarize  those  results  using  the  Assessment 
Data  Reporting  Template  shown  on  page  53. 

o  Finally,  the  department  may  choose  to  solicit  feedback  from  Air  Force  supervisors  about 
their  graduates'  research  and  writing  skills.  This  feedback  can  assist  department 
members  in  learning  how  well  cadets  are  applying  their  research  and  writing  skills  once 

In  some  departments,  it  may  not  be  tenable  to  have  all  faculty  grade  cadet  papers  using  an  agreed-upon  rubric. 
In  those  cases,  another  approach  would  be  to  have  faculty  members  grade  cadet  papers  as  they  normally  would, 
but  then  save  out  a  random  sample  of  the  papers  for  review  by  a  departmental  assessment  committee  at  the  end 
of  the  semester.  The  committee  could  review  those  papers  using  a  common  rubric  and  score  each  paper  using 
criteria  directly  relevant  to  the  outcome.  The  aggregated  results  could  then  be  summarized  using  the  Assessment 
Data  Reporting  Template  shown  on  page  47. 
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they  are  officers  on  active  duty.  As  was  the  case  with  the  feedback  from  cadets,  a  POC 
may  be  asked  to  summarize  this  supervisors'  feedback  using  the  Assessment  Data 
Reporting  Template  shown  on  page  53. 

•  Fourth,  the  program  will  need  to  establish  a  process  for  reviewing  and  making  sense  of  the 
assessment  data  collected  from  their  different  sources.  For  instance,  faculty  may  form  an 
"outcome  committee"  dedicated  specifically  to  examining  the  outcome  of  writing  effective 
research  papers.  That  committee  would  be  responsible  for  reviewing  the  summaries  submitted 
by  course  directors,  as  well  as  the  data  from  end-of-course  feedback,  focus  groups,  and  Air 
Force  supervisors.  They  may  be  asked  to  summarize  their  analyses  using  the  Outcome  Reporting 
Template  shown  on  page  61.  Note  that,  in  addition  to  summarizing  the  data,  this  template  also 
asks  for  their  recommendations  about  what  could  /  should  be  done  to  improve  cadets' 
accomplishment  of  the  outcome. 

•  Assume  for  a  moment  that  the  outcome  committee  discovers  that,  while  cadets  are  very  good 
at  summarizing  information  from  a  single  source,  they  have  difficulty  synthesizing  research 
material  from  multiple  sources.  Based  on  this  observation,  a  reasonable  suggestion  for 
improvement  would  be  to  provide  cadets  with  additional  assistance,  practice,  and  feedback  on 
this  skill.  Perhaps  that  could  be  done  by  inserting  an  additional  lesson  into  the  relevant  200-level 
course  about  how  to  synthesize  material  from  different  sources.  Perhaps  it  could  be  done  by 
adding  an  extra  assignment  in  one  of  the  required  courses  where  cadets  must  practice 
synthesizing  material  from  multiple  sources.  Or  perhaps  it  could  be  done  by  building  stronger 
connections  between  a  specific  course  and  the  resources  available  at  the  USAFA  Writing  Center. 
The  committee  will  be  charged  with  recommending  the  best  course  of  action  given  the  unique 
context  of  their  program,  but  the  key  is  to  turn  the  assessment  data  into  action.  And,  regardless 
of  what  action  faculty  decide  to  take,  it  is  a  good  idea  to  document  what  was  done  and  when 
any  changes  were  implemented.  That  way,  as  additional  assessment  data  are  collected  in  the 
future,  it  will  be  possible  to  determine  the  extent  to  which  the  change  had  the  desired  effect. 

One  possible  way  to  record  the  action  taken  to  improve  cadets'  outcome  accomplishment 
would  be  to  use  the  Sample  Outcome  improvement  Template,  printed  on  page  70. 

•  Implicit  in  this  discussion  is  that  the  program  will  indeed  engage  in  follow-up  assessment  to 
examine  whether  any  changes  were  effective.  So,  faculty  members  will  want  to  continue 
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examining  cadets'  ability  to  write  effective  research  papers  to  see  if  the  outcome  committee's 
recommended  changes  helped  solve  the  original  problem.  To  emphasize  the  importance  of  this 
follow-on  assessment,  the  last  item  on  the  Sample  Outcome  Improvement  Template  asks 
department  members  specifically  to  identify  how  and  when  the  follow-on  assessment  will  be 
taking  place,  In  addition,  the  idea  of  follow-on  assessment  is  included  in  the  fifth  (and  final) 
dimension  ("Using  Assessment  for  Improvement")  of  the  Academic  Program  Self-Assessment, 
shown  in  Table  9  on  page  72.  The  rubric  is  provided  in  its  entirety  in  Appendix  A. 
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SAMPLE  OUTCOME  IMPROVEMENT  TEMPLATE 


Rationale:  What  element(s)  of  the  outcome  is  most  in  need  of  improvement?  What  assessment 
evidence  do  we  have  showing  that  improvement  is  indeed  necessary? 


Description  of  Chaneelsl:  What  change(s)  will  we  be  implementing  to  improve  cadets'  achievement  of 
this  learning  outcome?  When  will  the  change(s)  be  taking  effect?  Why  do  we  anticipate  that  the  changes 
will  be  effective? 


Follow-On  Assessment  Plan:  How  will  we  reassess  this  outcome  to  determine  if  the  change  described 
above  had  a  beneficial  impact  on  cadets'  accomplishment  of  the  outcome?  When  will  that  follow-on 
assessment  take  place? 
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Document.  Document,  Document 

!  conciude  this  Handbook  with  one  final  urging:  Mease  keep  records  of  a//  the  great  assessment  work 
you  do  in  your  department,  This  includes  copies  of  your  cadet  learning  outcomes,  curriculum  maps, 
summaries  of  your  assessment  data,  as  well  as  records  of  changes  that  you  make  in  response  to  your 
results.  There  are  multiple  reasons  for  this,  the  most  important  of  which  is  to  maintain  continuity  in  your 
academic  program.  The  Air  Force  Academy  has  such  frequent  turnover  in  our  faculty  and  staff  that  it  is 
easy  for  departmental  best  practices  to  be  swept  away  if  they  are  not  clearly  documented.  Therefore, 
please  keep  track  of  what  you've  done.  And,  when  new  faculty  join  your  program,  !  encourage  you  to 
share  with  them  what  you've  done  as  wed.  That  way,  your  good  work  has  the  best  possible  chance  of 
living  on. 

Of  course,  there  is  institutional  incentive  for  you  to  keep  good  records  as  well.  Recall  that  USAFA  will  be 
expected  to  populate  an  electronic  "evidence  file"  that  documents  how  we  meet  the  criteria  for 
continued  Higher  Learning  Commission  accreditation.  As  such,  members  of  our  Institutional 
Accreditation  Team  will  be  asking  for  your  program  to  provide  evidence  that  it  is  successfully  applying 
the  various  steps  of  assessment  described  in  this  Handbook.  Obviously,  that  will  be  much  easier  for  you 
to  do  if  you  are  in  the  habit  of  keeping  those  records  for  yourselves  already. 

Finally,  I  encourage  you  to  share  your  assessment  documentation  on  our  Departmental  Assessment 
network  SharePoint  site:  https://eis.usafa.edu/academics/dean/dfe/DFED/dan/default.a5px.  The 

"Departmental  Assessment  Plans"  directory  contains  separate  folders  for  each  department  /  program  to 
share  their  assessment-related  materials.  By  keeping  your  folder  up-to-date,  you  will  help  your 
colleagues  in  around  USAFA  stay  in  touch  with  what  you  are  doing,  and  it  will  also  allow  all  of  us  to  learn 
from  each  other's  collective  experience.  As  programmatic  assessment  programs  continue  to  mature,  we 
can  also  use  the  material  on  the  SharePoint  site  to  make  updates  to  this  Assessment  Handbook.  That 
way,  this  Handbook  becomes  a  living  document  that  best  meets  the  needs  of  our  ever-improving  Air 
Force  Academy. 
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Table  9 

Academic  Program  Self-Assessment 
Using  Assessment  for  Improvement 


This  table  includes  one  row  of  the  broader  Academic  Program  Self-Assessment  rubric  that  has  been  in  place  within  DF.  (The  rubric  is  printed  in 
its  entirety  in  Appendix  A.)  in  conjunction  with  your  colleagues,  identify  which  description  (i.e.,  Level  0, 1,  2,  or  3)  best  matches  the  current  state 
of  cadet  learning  outcomes  within  your  academic  major  or  minor. 


Elements 

Question(s)  To  Think 
About 

Level  0 

Level  1 

Level  2 

Level  3 

Using 

Assessment 

for 

Improvement 

Does  my  department 
regularly  use 
assessment  results  for 
the  purposes  of 
improving  the  major? 

Is  the  department  in 
the  habit  of 
conducting  follow-up 
assessments  to  ensure 

that  the 
improvements 
actually  worked? 

Assessment 

results  have 
not  been 

used  for  the 
purposes  of 
improving 
the  major. 

There  is  at  least  one 
documented 
example  in  which 
assessment  results 
have  been  used  for 
the  purposes  of 
improving  the  major. 

Assessment  results  are 
regularly  used  for  the 
purposes  of  improving 
the  major.  However,  the 
department  is  not  in  the 
habit  of  performing 
follow-up  assessments 
to  ensure  that  the 
improvements  actually 
worked. 

Assessment  results  are 
regularly  used  for  the 
purposes  of  improving  the 
major.  The  department  is 
also  in  the  habit  of 
performing  follow-up 
assessments  to  ensure 
that  the  improvements 
actually  worked. 
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Appendix  A:  Academic  Program  Self-Assessment 


Department: 


Major: 


Directions:  For  each  row  in  the  rubric,  indicate  the  level  (0, 1,  2,  or  3)  that  most  accurately  describes  the  current  state  of  your  academic  program. 


Elements 

Question(s)  To  Think  About 

Level  0 

Level  1 

Level  2 

Level  3 

Cadet  Learning 
Outcomes 

Has  my  department  written  cadet 
learning  outcomes  for  this  major  in  a 
way  that  allows  for  meaningful 
assessment? 

Cadet  learning 
outcomes  have  not 
been  written  for  this 
major. 

Cadet  learning  outcomes  have  been 
written  for  this  major.  However,  they 
are  not  written  in  terms  of  what 
students  will  know  or  be  able  to  do  as 
a  result  of  being  in  the  major. 

Cadet  learning  outcomes  have  been  written 
in  terms  of  what  cadets  will  know  or  be  able 
to  do  as  a  result  of  being  in  the  major. 
However,  modest  improvements  are  needed 
because  one  of  the  following  is  true: 

1.  Outcomes  are  not  actionable  (i.e,,  they 
don't  include  an  action  verb), 

2.  Outcomes  are  not  clear  and 
understandable  to  faculty  and  cadets. 

Cadet  learning  outcomes  have  been  written  for 
this  major  in  a  way  that  allows  for  meaningful 
assessment. 

1,  Outcomes  are  written  in  terms  of  what 
cadets  will  know  or  be  able  to  do  as  a 
result  of  being  in  the  major, 

2,  Outcomes  are  actionable  (Le.,  they 
include  action  verbs}, 

3,  Outcomes  are  dear  and  understandable 
to  both  faculty  and  cadets. 

Curriculum  Map 
(Note:  A 
"'curriculum 
map""  is  a  matrix 
that  shows  which 
courses  In  your 
major  are  aligned 
with  which  of 
your  learning 
outcomes.} 

Has  my  department  developed  a 
curriculum  map  showing  dear  linkages 
between  cadet  learning  outcomes  and 
the  learning  experiences  (to  include 
coursework)  in  the  major? 

No  curriculum  map 
has  been  developed. 

A  curriculum  map  has  been 
developed.  However,  significant 
improvements  are  needed  because  of 
one  or  both  of  the  following: 

1.  There  are  one  or  more 
required  courses  that  do  not 
map  to  any  outcomes. 

2.  There  are  one  or  more 

outcomes  that  are  not 
addressed  in  the  curriculum. 

A  curriculum  map  has  been  developed, 
showing  clear  linkages  between  most  cadet 
learning  outcomes  and  the  teaming 
experiences  (to  include  coursework) 
provided  in  the  major.  However,  modest 
improvements  are  needed  because  one  or 
more  outcomes  lack  adequate  coverage, 

A  curriculum  map  has  been  developed,  showing 
dear  linkages  between  all  cadet  learning 
outcomes  and  the  learning  experiences  (to 
include  coursework)  in  the  major. 

Assessment 

Methods 

is  my  department  making  efforts  to 
assess  each  of  the  cadet  learning 
outcomes  in  the  major?  To  what  extent 
are  we  retying  on  direct  vs,  indirect 
measures? 

No  efforts  are  being 
made  to  assess  cadet 
learning  outcomes. 

The  department  is  beginning  to 
assess  at  feast  some  cadet  learning 
outcomes.  However,  assessment 
efforts  rely  completely  on  indirect 
measures  of  learning  (e,g.,  cadet 
feedback,  surveys,  focus  groups,  etc.). 

Assessment  of  cadet  learning  outcomes  is 
underway  but  not  fully  implemented.  At 
least  one  of  the  outcomes  incorporates  the 
use  of  direct  measures  of  learning  (eg., 
exams,  papers,  projects,  portfolios,  etc.). 

Department  is  fully  engaged  in  assessing  each  of 
the  cadet  learning  outcomes.  Direct  measures  of 
learning  (e.g.,  exams,  papers,  projects, 
portfolios,  etc.)  are  used  and  are  supplemented 
by  indirect  measures  (e,g,,  cadet  feedback, 
surveys,  focus  groups,  etc.),  as  appropriate. 

Making  Sense  of 
the  Results 

Has  my  department  established 
processes  for  compiling  and  making 
sense  of  assessment  results  for  each 
learning  outcome? 

No  efforts  are  being 
made  to  compile  or 
make  sense  of 
assessment  results. 

Assessment  results  are  compiled  from 
relevant  sources  (e.g.,  in  the  form  of 
course  diaries  or  course  director 
memoranda},  but  no  one  reviews  the 
compilation  as  a  way  to  better 
understand  how  well  cadets  are 
achieving  the  learning  outcomes. 

Assessment  results  are  compiled  from 
relevant  sources,  and  the  department  has 
begun  to  create  a  process  whereby  faculty 
review  those  results  for  a  better 
understanding  of  how  well  cadets  are 
achieving  at  least  one  learning  outcome. 

Assessment  results  are  compiled  from  relevant 
sources,  and  the  department  has  established  a 
process  whereby  faculty  review  those  results  for 
a  better  understanding  of  how  well  cadets  are 
achieving  each  of  the  learning  outcomes. 

Making  Sense  of 
the  Data 

Has  my  department  established 
processes  for  compiling  and  making 
sense  of  assessment  data  for  each 
learning  outcome? 

No  efforts  are  being 
made  to  compile  or 
make  sense  of 
assessment  data. 

Assessment  data  are  compiled  from 
relevant  sources  (e.g.,  in  the  form  of 
course  diaries  or  course  director 
memoranda),  but  no  one  reviews  the 
compilation  as  a  way  to  better 
understand  how  well  cadets  are 
achieving  the  learning  outcomes, 

Assessment  data  are  compiled  from  relevant 
sources,  and  the  department  has  begun  to 
create  a  process  whereby  faculty  review 
those  data  for  a  better  understanding  of 
how  well  cadets  are  achieving  feast  one 
learning  outcome. 

Assessment  data  are  compiled  from  relevant 
sources,  and  the  department  has  established  a 
process  whereby  faculty  review  those  data  for  a 
better  understanding  of  how  well  cadets  are 
achieving  each  of  the  learning  outcomes. 

Using 

Assessment  for 
Improvement 

Does  my  department  regularly  use 
assessment  results  for  the  purposes  of 
improving  the  major?  Is  the  department 
in  the  habit  of  conducting  follow-up 
assessments  to  ensure  that  the 
improvements  actually  worked? 

Assessment  results 
have  not  been  used 
for  the  purposes  of 
improving  the  major. 

There  is  at  least  one  documented 
example  in  which  assessment  results 
have  been  used  for  the  purposes  of 
improving  the  major. 

Assessment  results  are  regularly  used  for  the  1 
purposes  of  improving  the  major.  However, 
the  department  is  not  in  the  habit  of 
performing  follow- up  assessments  to  ensure 
that  the  improvements  actually  worked. 

Assessment  results  are  regularly  used  for  the 
purposes  of  improving  the  major.  The 
department  is  also  in  the  habit  of  performing 
follow-up  assessments  to  ensure  that  the 
improvements  actually  worked. 
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